__   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containers

Startup timestamp: 
Fri Feb  7 03:15:29 UTC 2025

Running all default registry services without migration
Running init script '/quay-registry/conf/init/certs_install.sh'
Installing extra certificates found in /quay-registry/conf/stack/extra_ca_certs directory
Running init script '/quay-registry/conf/init/client_certs.sh'
Running init script '/quay-registry/conf/init/copy_config_files.sh'
Running init script '/quay-registry/conf/init/d_validate_config_bundle.sh'
Validating Configuration
time="2025-02-07T03:15:30Z" level=debug msg="Validating AccessSettings"
time="2025-02-07T03:15:30Z" level=debug msg="Validating ActionLogArchiving"
time="2025-02-07T03:15:30Z" level=debug msg="Validating AppTokenAuthentication"
time="2025-02-07T03:15:30Z" level=debug msg="Validating AutoPrune"
time="2025-02-07T03:15:30Z" level=debug msg="Validating BitbucketBuildTrigger"
time="2025-02-07T03:15:30Z" level=debug msg="Validating BuildManager"
time="2025-02-07T03:15:30Z" level=debug msg="Validating Database"
time="2025-02-07T03:15:30Z" level=debug msg="Scheme: postgresql"
time="2025-02-07T03:15:30Z" level=debug msg="Host: quayregistry-quay-database:5432"
time="2025-02-07T03:15:30Z" level=debug msg="Db: quayregistry-quay-database"
time="2025-02-07T03:15:30Z" level=debug msg="Params: "
time="2025-02-07T03:15:30Z" level=debug msg="Including params "
time="2025-02-07T03:15:30Z" level=debug msg="Pinging database at hostname: quayregistry-quay-database:5432."
time="2025-02-07T03:15:30Z" level=debug msg="Database version: 13.18"
plpgsql
pg_trgm
time="2025-02-07T03:15:30Z" level=debug msg="Validating DistributedStorage"
time="2025-02-07T03:15:30Z" level=debug msg="Using IBM Cloud/ODF/RadosGW storage."
time="2025-02-07T03:15:30Z" level=debug msg="Storage parameters: "
time="2025-02-07T03:15:30Z" level=debug msg="hostname: s3.openshift-storage.svc.cluster.local:443, bucket name: quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360, TLS enabled: true"
time="2025-02-07T03:15:30Z" level=debug msg="Validating ElasticSearch"
time="2025-02-07T03:15:30Z" level=debug msg="Validating Email"
time="2025-02-07T03:15:30Z" level=debug msg="Validating GitHubBuildTrigger"
time="2025-02-07T03:15:30Z" level=debug msg="Validating GitHubLogin"
time="2025-02-07T03:15:30Z" level=debug msg="Validating GitLabBuildTrigger"
time="2025-02-07T03:15:30Z" level=debug msg="Validating GoogleLogin"
time="2025-02-07T03:15:30Z" level=debug msg="Validating HostSettings"
time="2025-02-07T03:15:30Z" level=debug msg="Validating JWTAuthentication"
time="2025-02-07T03:15:30Z" level=debug msg="Validating LDAP"
time="2025-02-07T03:15:30Z" level=debug msg="Validating OIDC"
time="2025-02-07T03:15:30Z" level=debug msg="Validating QuayDocumentation"
time="2025-02-07T03:15:30Z" level=debug msg="Validating Redis"
time="2025-02-07T03:15:30Z" level=debug msg="Address: quayregistry-quay-redis:6379"
time="2025-02-07T03:15:30Z" level=debug msg="Username: "
time="2025-02-07T03:15:30Z" level=debug msg="Password Len: 0"
time="2025-02-07T03:15:30Z" level=debug msg="Ssl: <nil>"
time="2025-02-07T03:15:30Z" level=debug msg="Address: quayregistry-quay-redis:6379"
time="2025-02-07T03:15:30Z" level=debug msg="Username: "
time="2025-02-07T03:15:30Z" level=debug msg="Password Len: 0"
time="2025-02-07T03:15:30Z" level=debug msg="Ssl: <nil>"
time="2025-02-07T03:15:30Z" level=debug msg="Validating RepoMirror"
time="2025-02-07T03:15:30Z" level=debug msg="Validating SecurityScanner"
time="2025-02-07T03:15:30Z" level=debug msg="Validating TeamSyncing"
time="2025-02-07T03:15:30Z" level=debug msg="Validating TimeMachine"
time="2025-02-07T03:15:30Z" level=debug msg="Validating UserVisibleSettings"
+------------------------+-------+--------+
|      Field Group       | Error | Status |
+------------------------+-------+--------+
| AccessSettings         | -     | 🟢     |
+------------------------+-------+--------+
| ActionLogArchiving     | -     | 🟢     |
+------------------------+-------+--------+
| AppTokenAuthentication | -     | 🟢     |
+------------------------+-------+--------+
| AutoPrune              | -     | 🟢     |
+------------------------+-------+--------+
| BitbucketBuildTrigger  | -     | 🟢     |
+------------------------+-------+--------+
| BuildManager           | -     | 🟢     |
+------------------------+-------+--------+
| Database               | -     | 🟢     |
+------------------------+-------+--------+
| DistributedStorage     | -     | 🟢     |
+------------------------+-------+--------+
| ElasticSearch          | -     | 🟢     |
+------------------------+-------+--------+
| Email                  | -     | 🟢     |
+------------------------+-------+--------+
| GitHubBuildTrigger     | -     | 🟢     |
+------------------------+-------+--------+
| GitHubLogin            | -     | 🟢     |
+------------------------+-------+--------+
| GitLabBuildTrigger     | -     | 🟢     |
+------------------------+-------+--------+
| GoogleLogin            | -     | 🟢     |
+------------------------+-------+--------+
| HostSettings           | -     | 🟢     |
+------------------------+-------+--------+
| JWTAuthentication      | -     | 🟢     |
+------------------------+-------+--------+
| LDAP                   | -     | 🟢     |
+------------------------+-------+--------+
| OIDC                   | -     | 🟢     |
+------------------------+-------+--------+
| QuayDocumentation      | -     | 🟢     |
+------------------------+-------+--------+
| Redis                  | -     | 🟢     |
+------------------------+-------+--------+
| RepoMirror             | -     | 🟢     |
+------------------------+-------+--------+
| SecurityScanner        | -     | 🟢     |
+------------------------+-------+--------+
| TeamSyncing            | -     | 🟢     |
+------------------------+-------+--------+
| TimeMachine            | -     | 🟢     |
+------------------------+-------+--------+
| UserVisibleSettings    | -     | 🟢     |
+------------------------+-------+--------+
Running init script '/quay-registry/conf/init/nginx_conf_create.sh'
Running init script '/quay-registry/conf/init/supervisord_conf_create.sh'
Running init script '/quay-registry/conf/init/zz_boot.sh'
2025-02-07 03:15:32,973 INFO RPC interface 'supervisor' initialized
2025-02-07 03:15:32,973 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-02-07 03:15:32,973 INFO supervisord started with pid 7
2025-02-07 03:15:33,975 INFO spawned: 'stdout' with pid 53
2025-02-07 03:15:33,977 INFO spawned: 'autopruneworker' with pid 54
2025-02-07 03:15:33,978 INFO spawned: 'blobuploadcleanupworker' with pid 55
2025-02-07 03:15:33,979 INFO spawned: 'builder' with pid 56
2025-02-07 03:15:33,981 INFO spawned: 'buildlogsarchiver' with pid 57
2025-02-07 03:15:33,983 INFO spawned: 'chunkcleanupworker' with pid 58
2025-02-07 03:15:33,984 INFO spawned: 'dnsmasq' with pid 59
2025-02-07 03:15:33,986 INFO spawned: 'expiredappspecifictokenworker' with pid 60
2025-02-07 03:15:33,988 INFO spawned: 'exportactionlogsworker' with pid 61
2025-02-07 03:15:33,989 INFO spawned: 'gcworker' with pid 62
2025-02-07 03:15:33,991 INFO spawned: 'globalpromstats' with pid 63
2025-02-07 03:15:33,993 INFO spawned: 'gunicorn-registry' with pid 64
2025-02-07 03:15:33,995 INFO spawned: 'gunicorn-secscan' with pid 65
2025-02-07 03:15:33,997 INFO spawned: 'gunicorn-web' with pid 66
2025-02-07 03:15:33,998 INFO spawned: 'logrotateworker' with pid 67
2025-02-07 03:15:34,000 INFO spawned: 'manifestbackfillworker' with pid 68
2025-02-07 03:15:34,002 INFO spawned: 'manifestsubjectbackfillworker' with pid 69
2025-02-07 03:15:34,012 INFO spawned: 'memcache' with pid 70
2025-02-07 03:15:34,014 INFO spawned: 'namespacegcworker' with pid 71
2025-02-07 03:15:34,016 INFO spawned: 'nginx' with pid 72
2025-02-07 03:15:34,018 INFO spawned: 'notificationworker' with pid 73
2025-02-07 03:15:34,019 INFO spawned: 'pushgateway' with pid 74
2025-02-07 03:15:34,021 INFO spawned: 'queuecleanupworker' with pid 75
2025-02-07 03:15:34,023 INFO spawned: 'quotaregistrysizeworker' with pid 76
2025-02-07 03:15:34,025 INFO spawned: 'quotatotalworker' with pid 77
2025-02-07 03:15:34,034 INFO spawned: 'reconciliationworker' with pid 83
2025-02-07 03:15:34,114 INFO spawned: 'repositoryactioncounter' with pid 84
2025-02-07 03:15:34,116 INFO spawned: 'repositorygcworker' with pid 93
2025-02-07 03:15:34,120 INFO spawned: 'securityscanningnotificationworker' with pid 97
2025-02-07 03:15:34,124 INFO spawned: 'securityworker' with pid 98
2025-02-07 03:15:34,126 INFO spawned: 'servicekey' with pid 99
2025-02-07 03:15:34,128 INFO spawned: 'storagereplication' with pid 100
2025-02-07 03:15:34,130 INFO spawned: 'teamsyncworker' with pid 103
2025-02-07 03:15:35,220 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: autopruneworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: blobuploadcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: builder entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: buildlogsarchiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: chunkcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: dnsmasq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: expiredappspecifictokenworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: exportactionlogsworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: gcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: globalpromstats entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: gunicorn-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,220 INFO success: gunicorn-secscan entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: gunicorn-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: logrotateworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: manifestbackfillworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: manifestsubjectbackfillworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: memcache entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: namespacegcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: notificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: pushgateway entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: queuecleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: quotaregistrysizeworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: quotatotalworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: reconciliationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: repositoryactioncounter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: repositorygcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: securityscanningnotificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: securityworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: servicekey entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: storagereplication entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-07 03:15:35,221 INFO success: teamsyncworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
dnsmasq stderr | dnsmasq: started, version 2.79 cachesize 150
dnsmasq stderr | dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
dnsmasq stderr | dnsmasq: reading /etc/resolv.conf
dnsmasq stderr | dnsmasq: using nameserver 172.30.0.10#53
dnsmasq stderr | dnsmasq: read /etc/hosts - 7 addresses
nginx stdout | 2025/02/07 03:15:34 [alert] 91#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2025/02/07 03:15:34 [alert] 87#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2025/02/07 03:15:34 [alert] 89#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2025/02/07 03:15:34 [alert] 90#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2025/02/07 03:15:34 [alert] 85#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2025/02/07 03:15:34 [alert] 88#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2025/02/07 03:15:34 [alert] 86#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2025/02/07 03:15:34 [alert] 92#0: setpriority(-10) failed (13: Permission denied)
pushgateway stderr | ts=2025-02-07T03:15:34.217Z caller=main.go:86 level=info msg="starting pushgateway" version="(version=, branch=, revision=unknown)"
pushgateway stderr | ts=2025-02-07T03:15:34.217Z caller=main.go:87 level=info build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=, date=, tags=strictfipsruntime)"
pushgateway stderr | ts=2025-02-07T03:15:34.218Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9091
pushgateway stderr | ts=2025-02-07T03:15:34.218Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9091
nginx stdout | 2025/02/07 03:15:44 [crit] 91#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.129.2.36:8080"
nginx stdout | 2025/02/07 03:15:44 [crit] 91#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.129.2.36:8080"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:15:44 +0000] "GET /health/instance HTTP/1.1" 502 157 "-" "kube-probe/1.30" (0.000 118 0.000 : 0.000)
nginx stdout | 2025/02/07 03:15:59 [crit] 87#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.129.2.36:8080"
nginx stdout | 2025/02/07 03:15:59 [crit] 87#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.129.2.36:8080"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:15:59 +0000] "GET /health/instance HTTP/1.1" 502 157 "-" "kube-probe/1.30" (0.000 118 0.000 : 0.000)
queuecleanupworker stdout | 2025-02-07 03:15:59,318 [75] [DEBUG] [workers.worker] Scheduling worker.
queuecleanupworker stdout | 2025-02-07 03:15:59,319 [75] [INFO] [apscheduler.scheduler] Scheduler started
queuecleanupworker stdout | 2025-02-07 03:15:59,517 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
queuecleanupworker stdout | 2025-02-07 03:15:59,517 [75] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 10:32:39.518525
queuecleanupworker stdout | 2025-02-07 03:15:59,519 [75] [INFO] [apscheduler.scheduler] Added job "QueueCleanupWorker._cleanup_queue" to job store "default"
queuecleanupworker stdout | 2025-02-07 03:15:59,519 [75] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
queuecleanupworker stdout | 2025-02-07 03:15:59,521 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
queuecleanupworker stdout | 2025-02-07 03:15:59,522 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 10:32:39.518525+00:00 (in 26199.996250 seconds)
repositoryactioncounter stdout | 2025-02-07 03:15:59,731 [84] [DEBUG] [workers.worker] Scheduling worker.
repositoryactioncounter stdout | 2025-02-07 03:15:59,731 [84] [INFO] [apscheduler.scheduler] Scheduler started
repositoryactioncounter stdout | 2025-02-07 03:15:59,812 [84] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositoryactioncounter stdout | 2025-02-07 03:15:59,812 [84] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 07:15:44.813292
repositoryactioncounter stdout | 2025-02-07 03:15:59,813 [84] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
repositoryactioncounter stdout | 2025-02-07 03:15:59,815 [84] [INFO] [apscheduler.scheduler] Added job "RepositoryActionCountWorker._run_counting" to job store "default"
repositoryactioncounter stdout | 2025-02-07 03:15:59,816 [84] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositoryactioncounter stdout | 2025-02-07 03:15:59,816 [84] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 07:15:44.813292+00:00 (in 14384.996562 seconds)
storagereplication stdout | 2025-02-07 03:15:59,933 [100] [DEBUG] [__main__] Full storage replication disabled; skipping
teamsyncworker stdout | 2025-02-07 03:16:03,120 [103] [DEBUG] [__main__] Team syncing is disabled; sleeping
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,115 [76] [DEBUG] [workers.worker] Scheduling worker.
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,116 [76] [INFO] [apscheduler.scheduler] Scheduler started
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,117 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,117 [76] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:56.118855
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,119 [76] [INFO] [apscheduler.scheduler] Added job "QuotaRegistrySizeWorker._calculate_registry_size" to job store "default"
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,118 [76] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,120 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:16:04,121 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:56.118855+00:00 (in 51.997449 seconds)
logrotateworker stdout | 2025-02-07 03:16:09,623 [67] [DEBUG] [__main__] Action log rotation worker not enabled; skipping
builder stderr | /app/lib/python3.9/site-packages/google/protobuf/runtime_version.py:112: UserWarning: Protobuf gencode version 5.27.2 is older than the runtime version 5.28.2 at buildman.proto. Please avoid checked-in Protobuf gencode that can be obsolete.
builder stderr |   warnings.warn(
builder stdout | 2025-02-07 03:16:09,829 [56] [DEBUG] [__main__] Building is disabled. Please enable the feature flag
notificationworker stdout | 2025-02-07 03:16:10,718 [73] [DEBUG] [workers.worker] Scheduling worker.
notificationworker stdout | 2025-02-07 03:16:10,720 [73] [INFO] [apscheduler.scheduler] Scheduler started
notificationworker stdout | 2025-02-07 03:16:10,731 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:10,731 [73] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:18.732147
notificationworker stdout | 2025-02-07 03:16:10,732 [73] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
notificationworker stdout | 2025-02-07 03:16:10,732 [73] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
notificationworker stdout | 2025-02-07 03:16:10,813 [73] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:17:33.814166
notificationworker stdout | 2025-02-07 03:16:10,813 [73] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
notificationworker stdout | 2025-02-07 03:16:10,813 [73] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:50.814677
notificationworker stdout | 2025-02-07 03:16:10,813 [73] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
notificationworker stdout | 2025-02-07 03:16:10,813 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:10,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:18.732147+00:00 (in 7.918136 seconds)
notificationworker stdout | 2025-02-07 03:16:10,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:10,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:18.732147+00:00 (in 7.918037 seconds)
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,718 [60] [DEBUG] [__main__] Starting expired app specific token GC worker
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,719 [60] [DEBUG] [__main__] Found expiration window: 1d
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,722 [60] [DEBUG] [workers.worker] Scheduling worker.
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,724 [60] [INFO] [apscheduler.scheduler] Scheduler started
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,814 [60] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,814 [60] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:56:00.815400
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,816 [60] [INFO] [apscheduler.scheduler] Added job "ExpiredAppSpecificTokenWorker._gc_expired_tokens" to job store "default"
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,815 [60] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,822 [60] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2025-02-07 03:16:11,823 [60] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:00.815400+00:00 (in 2388.992348 seconds)
chunkcleanupworker stdout | 2025-02-07 03:16:12,821 [58] [DEBUG] [__main__] Swift storage not detected; sleeping
servicekey stdout | 2025-02-07 03:16:13,229 [99] [DEBUG] [workers.worker] Scheduling worker.
servicekey stdout | 2025-02-07 03:16:13,231 [99] [INFO] [apscheduler.scheduler] Scheduler started
servicekey stdout | 2025-02-07 03:16:13,324 [99] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2025-02-07 03:16:13,324 [99] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 04:03:55.325482
servicekey stdout | 2025-02-07 03:16:13,326 [99] [INFO] [apscheduler.scheduler] Added job "ServiceKeyWorker._refresh_service_key" to job store "default"
servicekey stdout | 2025-02-07 03:16:13,325 [99] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
servicekey stdout | 2025-02-07 03:16:13,327 [99] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2025-02-07 03:16:13,328 [99] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:55.325482+00:00 (in 2861.996990 seconds)
globalpromstats stdout | 2025-02-07 03:16:13,717 [63] [DEBUG] [workers.worker] Scheduling worker.
globalpromstats stdout | 2025-02-07 03:16:13,718 [63] [INFO] [apscheduler.scheduler] Scheduler started
autopruneworker stdout | 2025-02-07 03:16:13,731 [54] [DEBUG] [workers.worker] Scheduling worker.
autopruneworker stdout | 2025-02-07 03:16:13,732 [54] [INFO] [apscheduler.scheduler] Scheduler started
globalpromstats stdout | 2025-02-07 03:16:13,734 [63] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2025-02-07 03:16:13,734 [63] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:48:24.735445
globalpromstats stdout | 2025-02-07 03:16:13,735 [63] [INFO] [apscheduler.scheduler] Added job "GlobalPrometheusStatsWorker._try_report_stats" to job store "default"
globalpromstats stdout | 2025-02-07 03:16:13,735 [63] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
globalpromstats stdout | 2025-02-07 03:16:13,736 [63] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2025-02-07 03:16:13,736 [63] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:24.735445+00:00 (in 1930.998790 seconds)
autopruneworker stdout | 2025-02-07 03:16:13,830 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:16:13,830 [54] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:41.831521
autopruneworker stdout | 2025-02-07 03:16:13,832 [54] [INFO] [apscheduler.scheduler] Added job "AutoPruneWorker.prune" to job store "default"
autopruneworker stdout | 2025-02-07 03:16:13,831 [54] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
autopruneworker stdout | 2025-02-07 03:16:13,837 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:16:13,837 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:41.831521+00:00 (in 27.994089 seconds)
securityscanningnotificationworker stdout | 2025-02-07 03:16:14,127 [97] [DEBUG] [__main__] Security scanner disabled; sleeping
manifestbackfillworker stdout | 2025-02-07 03:16:14,130 [68] [DEBUG] [workers.worker] Scheduling worker.
manifestbackfillworker stdout | 2025-02-07 03:16:14,132 [68] [INFO] [apscheduler.scheduler] Scheduler started
manifestbackfillworker stdout | 2025-02-07 03:16:14,133 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2025-02-07 03:16:14,133 [68] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:41:52.134599
manifestbackfillworker stdout | 2025-02-07 03:16:14,135 [68] [INFO] [apscheduler.scheduler] Added job "ManifestBackfillWorker._backfill_manifests" to job store "default"
manifestbackfillworker stdout | 2025-02-07 03:16:14,134 [68] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
manifestbackfillworker stdout | 2025-02-07 03:16:14,135 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2025-02-07 03:16:14,135 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:52.134599+00:00 (in 1537.998934 seconds)
nginx stdout | 2025/02/07 03:16:14 [crit] 89#0: *7 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.129.2.36:8080"
nginx stdout | 2025/02/07 03:16:14 [crit] 89#0: *7 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.129.2.36:8080"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:16:14 +0000] "GET /health/instance HTTP/1.1" 502 157 "-" "kube-probe/1.30" (0.000 118 0.000 : 0.000)
gcworker stdout | 2025-02-07 03:16:14,331 [62] [DEBUG] [workers.worker] Scheduling worker.
gcworker stdout | 2025-02-07 03:16:14,333 [62] [INFO] [apscheduler.scheduler] Scheduler started
repositorygcworker stdout | 2025-02-07 03:16:14,336 [93] [DEBUG] [__main__] Starting repository GC worker
repositorygcworker stdout | 2025-02-07 03:16:14,414 [93] [DEBUG] [workers.worker] Scheduling worker.
repositorygcworker stdout | 2025-02-07 03:16:14,416 [93] [INFO] [apscheduler.scheduler] Scheduler started
repositorygcworker stdout | 2025-02-07 03:16:14,418 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:16:14,418 [93] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:17:03.419110
repositorygcworker stdout | 2025-02-07 03:16:14,419 [93] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
repositorygcworker stdout | 2025-02-07 03:16:14,419 [93] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:18:32.420707
repositorygcworker stdout | 2025-02-07 03:16:14,419 [93] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
repositorygcworker stdout | 2025-02-07 03:16:14,419 [93] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:23.420886
repositorygcworker stdout | 2025-02-07 03:16:14,420 [93] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
repositorygcworker stdout | 2025-02-07 03:16:14,419 [93] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
repositorygcworker stdout | 2025-02-07 03:16:14,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:16:14,422 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:23.420886+00:00 (in 8.998857 seconds)
gcworker stdout | 2025-02-07 03:16:14,421 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:16:14,422 [62] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:24.423055
gcworker stdout | 2025-02-07 03:16:14,423 [62] [INFO] [apscheduler.scheduler] Added job "GarbageCollectionWorker._garbage_collection_repos" to job store "default"
gcworker stdout | 2025-02-07 03:16:14,423 [62] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:40.424250
gcworker stdout | 2025-02-07 03:16:14,423 [62] [INFO] [apscheduler.scheduler] Added job "GarbageCollectionWorker._scan_notifications" to job store "default"
gcworker stdout | 2025-02-07 03:16:14,422 [62] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
gcworker stdout | 2025-02-07 03:16:14,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:16:14,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:24.423055+00:00 (in 9.999417 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,633 [69] [DEBUG] [workers.worker] Scheduling worker.
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,635 [69] [INFO] [apscheduler.scheduler] Scheduler started
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,712 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,712 [69] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:56.713547
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,714 [69] [INFO] [apscheduler.scheduler] Added job "ManifestSubjectBackfillWorker._backfill_manifest_subject" to job store "default"
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,714 [69] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:37.715878
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,715 [69] [INFO] [apscheduler.scheduler] Added job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type" to job store "default"
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,714 [69] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:14,717 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:37.715878+00:00 (in 22.998377 seconds)
gunicorn-web stdout | 2025-02-07 03:16:15,522 [66] [DEBUG] [app] Loading default config.
gunicorn-web stdout | 2025-02-07 03:16:15,523 [66] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml
gunicorn-web stdout | 2025-02-07 03:16:15,534 [66] [DEBUG] [app] Loaded config
gunicorn-web stdout | 2025-02-07 03:16:15,536 [66] [INFO] [util.ipresolver] Loading AWS IP ranges from disk
namespacegcworker stdout | 2025-02-07 03:16:15,612 [71] [DEBUG] [__main__] Starting namespace GC worker
namespacegcworker stdout | 2025-02-07 03:16:15,614 [71] [DEBUG] [workers.worker] Scheduling worker.
gunicorn-web stdout | 2025-02-07 03:16:15,615 [66] [DEBUG] [util.ipresolver] Building AWS IP ranges
namespacegcworker stdout | 2025-02-07 03:16:15,616 [71] [INFO] [apscheduler.scheduler] Scheduler started
namespacegcworker stdout | 2025-02-07 03:16:15,619 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:16:15,620 [71] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
namespacegcworker stdout | 2025-02-07 03:16:15,620 [71] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:31.621131
namespacegcworker stdout | 2025-02-07 03:16:15,622 [71] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
namespacegcworker stdout | 2025-02-07 03:16:15,623 [71] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:17:13.624301
namespacegcworker stdout | 2025-02-07 03:16:15,624 [71] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
namespacegcworker stdout | 2025-02-07 03:16:15,624 [71] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:32.625945
namespacegcworker stdout | 2025-02-07 03:16:15,625 [71] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
namespacegcworker stdout | 2025-02-07 03:16:15,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:16:15,627 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:31.621131+00:00 (in 15.993836 seconds)
namespacegcworker stdout | 2025-02-07 03:16:15,627 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:16:15,628 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:31.621131+00:00 (in 15.993096 seconds)
buildlogsarchiver stdout | 2025-02-07 03:16:15,917 [57] [DEBUG] [workers.worker] Scheduling worker.
buildlogsarchiver stdout | 2025-02-07 03:16:15,917 [57] [INFO] [apscheduler.scheduler] Scheduler started
buildlogsarchiver stdout | 2025-02-07 03:16:15,918 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:16:15,918 [57] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
buildlogsarchiver stdout | 2025-02-07 03:16:15,918 [57] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:33.919286
buildlogsarchiver stdout | 2025-02-07 03:16:15,919 [57] [INFO] [apscheduler.scheduler] Added job "ArchiveBuildLogsWorker._archive_redis_buildlogs" to job store "default"
buildlogsarchiver stdout | 2025-02-07 03:16:15,920 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:16:15,920 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:33.919286+00:00 (in 17.999171 seconds)
exportactionlogsworker stdout | 2025-02-07 03:16:15,932 [61] [DEBUG] [__main__] Starting export action logs worker
exportactionlogsworker stdout | 2025-02-07 03:16:15,934 [61] [DEBUG] [workers.worker] Scheduling worker.
exportactionlogsworker stdout | 2025-02-07 03:16:15,935 [61] [INFO] [apscheduler.scheduler] Scheduler started
exportactionlogsworker stdout | 2025-02-07 03:16:15,935 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:16:15,935 [61] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
exportactionlogsworker stdout | 2025-02-07 03:16:15,935 [61] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:17:04.936688
exportactionlogsworker stdout | 2025-02-07 03:16:15,936 [61] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
exportactionlogsworker stdout | 2025-02-07 03:16:15,936 [61] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:19:52.937648
exportactionlogsworker stdout | 2025-02-07 03:16:15,936 [61] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
exportactionlogsworker stdout | 2025-02-07 03:16:15,936 [61] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:16:59.937899
exportactionlogsworker stdout | 2025-02-07 03:16:15,937 [61] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
exportactionlogsworker stdout | 2025-02-07 03:16:15,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:16:15,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:59.937899+00:00 (in 44.000660 seconds)
gunicorn-web stdout | 2025-02-07 03:16:15,946 [66] [DEBUG] [util.ipresolver] Finished building AWS IP ranges
gunicorn-web stdout | 2025-02-07 03:16:15,948 [66] [DEBUG] [botocore.hooks] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
gunicorn-web stdout | 2025-02-07 03:16:15,949 [66] [DEBUG] [botocore.hooks] Changing event name from before-call.apigateway to before-call.api-gateway
gunicorn-web stdout | 2025-02-07 03:16:16,012 [66] [DEBUG] [botocore.hooks] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
gunicorn-web stdout | 2025-02-07 03:16:16,013 [66] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
gunicorn-web stdout | 2025-02-07 03:16:16,014 [66] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
gunicorn-web stdout | 2025-02-07 03:16:16,014 [66] [DEBUG] [botocore.hooks] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
gunicorn-web stdout | 2025-02-07 03:16:16,015 [66] [DEBUG] [botocore.hooks] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
gunicorn-web stdout | 2025-02-07 03:16:16,018 [66] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
gunicorn-web stdout | 2025-02-07 03:16:16,018 [66] [DEBUG] [botocore.hooks] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
gunicorn-web stdout | 2025-02-07 03:16:16,018 [66] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
gunicorn-web stdout | 2025-02-07 03:16:16,018 [66] [DEBUG] [botocore.hooks] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,054 [55] [DEBUG] [workers.worker] Scheduling worker.
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,055 [55] [INFO] [apscheduler.scheduler] Scheduler started
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,056 [55] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,056 [55] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,056 [55] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:31:05.057762
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,057 [55] [INFO] [apscheduler.scheduler] Added job "BlobUploadCleanupWorker._try_cleanup_uploads" to job store "default"
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,057 [55] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2025-02-07 03:16:16,057 [55] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:05.057762+00:00 (in 888.999878 seconds)
gunicorn-web stdout | 2025-02-07 03:16:16,122 [66] [DEBUG] [data.database] Configuring database
gunicorn-web stdout | 2025-02-07 03:16:16,123 [66] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:16:16,123 [66] [INFO] [data.secscan_model] ===============================
gunicorn-web stdout | 2025-02-07 03:16:16,123 [66] [INFO] [data.secscan_model] Using split secscan model: `[<data.secscan_model.secscan_v4_model.NoopV4SecurityScanner object at 0x7f163f29e100>]`
gunicorn-web stdout | 2025-02-07 03:16:16,123 [66] [INFO] [data.secscan_model] ===============================
gunicorn-web stdout | 2025-02-07 03:16:16,124 [66] [DEBUG] [data.logs_model] Configuring log model
gunicorn-web stdout | 2025-02-07 03:16:16,124 [66] [INFO] [data.logs_model] ===============================
gunicorn-web stdout | 2025-02-07 03:16:16,124 [66] [INFO] [data.logs_model] Using logs model `<data.logs_model.table_logs_model.TableLogsModel object at 0x7f163f29e220>`
gunicorn-web stdout | 2025-02-07 03:16:16,124 [66] [INFO] [data.logs_model] ===============================
gunicorn-secscan stdout | 2025-02-07 03:16:16,560 [65] [DEBUG] [app] Loading default config.
gunicorn-secscan stdout | 2025-02-07 03:16:16,560 [65] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml
gunicorn-secscan stdout | 2025-02-07 03:16:16,566 [65] [DEBUG] [app] Loaded config
gunicorn-secscan stdout | 2025-02-07 03:16:16,567 [65] [INFO] [util.ipresolver] Loading AWS IP ranges from disk
gunicorn-secscan stdout | 2025-02-07 03:16:16,574 [65] [DEBUG] [util.ipresolver] Building AWS IP ranges
gunicorn-secscan stdout | 2025-02-07 03:16:16,675 [65] [DEBUG] [util.ipresolver] Finished building AWS IP ranges
gunicorn-secscan stdout | 2025-02-07 03:16:16,677 [65] [DEBUG] [botocore.hooks] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
gunicorn-secscan stdout | 2025-02-07 03:16:16,678 [65] [DEBUG] [botocore.hooks] Changing event name from before-call.apigateway to before-call.api-gateway
gunicorn-secscan stdout | 2025-02-07 03:16:16,678 [65] [DEBUG] [botocore.hooks] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
gunicorn-secscan stdout | 2025-02-07 03:16:16,679 [65] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
gunicorn-secscan stdout | 2025-02-07 03:16:16,679 [65] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
gunicorn-secscan stdout | 2025-02-07 03:16:16,712 [65] [DEBUG] [botocore.hooks] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
gunicorn-secscan stdout | 2025-02-07 03:16:16,712 [65] [DEBUG] [botocore.hooks] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
gunicorn-secscan stdout | 2025-02-07 03:16:16,714 [65] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
gunicorn-secscan stdout | 2025-02-07 03:16:16,714 [65] [DEBUG] [botocore.hooks] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
gunicorn-secscan stdout | 2025-02-07 03:16:16,714 [65] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
gunicorn-secscan stdout | 2025-02-07 03:16:16,714 [65] [DEBUG] [botocore.hooks] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
gunicorn-secscan stdout | 2025-02-07 03:16:16,744 [65] [DEBUG] [data.database] Configuring database
gunicorn-secscan stdout | 2025-02-07 03:16:16,744 [65] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-secscan stdout | 2025-02-07 03:16:16,745 [65] [INFO] [data.secscan_model] ===============================
gunicorn-secscan stdout | 2025-02-07 03:16:16,745 [65] [INFO] [data.secscan_model] Using split secscan model: `[<data.secscan_model.secscan_v4_model.NoopV4SecurityScanner object at 0x7fb681621bb0>]`
gunicorn-secscan stdout | 2025-02-07 03:16:16,745 [65] [INFO] [data.secscan_model] ===============================
gunicorn-secscan stdout | 2025-02-07 03:16:16,745 [65] [DEBUG] [data.logs_model] Configuring log model
gunicorn-secscan stdout | 2025-02-07 03:16:16,745 [65] [INFO] [data.logs_model] ===============================
gunicorn-secscan stdout | 2025-02-07 03:16:16,745 [65] [INFO] [data.logs_model] Using logs model `<data.logs_model.table_logs_model.TableLogsModel object at 0x7fb6816214f0>`
gunicorn-secscan stdout | 2025-02-07 03:16:16,745 [65] [INFO] [data.logs_model] ===============================
gunicorn-secscan stdout | 2025-02-07 03:16:16,956 [65] [DEBUG] [__config__] Starting secscan gunicorn with 2 workers and gevent worker class
gunicorn-secscan stderr | Traceback (most recent call last):
gunicorn-secscan stderr |   File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
gunicorn-secscan stderr |   File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
gunicorn-secscan stderr | AssertionError: (None, <callback at 0x7fb68159dbc0 args=([],)>)
gunicorn-secscan stderr | 2025-02-07T03:16:16Z 
gunicorn-secscan stderr | <callback at 0x7fb68159dbc0 args=([],)> failed with AssertionError
gunicorn-web stdout | 2025-02-07 03:16:17,267 [66] [DEBUG] [__config__] Starting web gunicorn with 4 workers and gevent worker class
gunicorn-web stderr | Traceback (most recent call last):
gunicorn-web stderr |   File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
gunicorn-web stderr |   File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
gunicorn-web stderr | AssertionError: (None, <callback at 0x7f163fdaee80 args=([],)>)
gunicorn-web stderr | 2025-02-07T03:16:17Z 
gunicorn-web stderr | <callback at 0x7f163fdaee80 args=([],)> failed with AssertionError
gunicorn-registry stdout | 2025-02-07 03:16:17,447 [64] [DEBUG] [app] Loading default config.
gunicorn-registry stdout | 2025-02-07 03:16:17,448 [64] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml
gunicorn-registry stdout | 2025-02-07 03:16:17,453 [64] [DEBUG] [app] Loaded config
gunicorn-registry stdout | 2025-02-07 03:16:17,454 [64] [INFO] [util.ipresolver] Loading AWS IP ranges from disk
gunicorn-registry stdout | 2025-02-07 03:16:17,461 [64] [DEBUG] [util.ipresolver] Building AWS IP ranges
gunicorn-registry stdout | 2025-02-07 03:16:17,530 [64] [DEBUG] [util.ipresolver] Finished building AWS IP ranges
gunicorn-registry stdout | 2025-02-07 03:16:17,532 [64] [DEBUG] [botocore.hooks] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
gunicorn-registry stdout | 2025-02-07 03:16:17,533 [64] [DEBUG] [botocore.hooks] Changing event name from before-call.apigateway to before-call.api-gateway
gunicorn-registry stdout | 2025-02-07 03:16:17,533 [64] [DEBUG] [botocore.hooks] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
gunicorn-registry stdout | 2025-02-07 03:16:17,534 [64] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
gunicorn-registry stdout | 2025-02-07 03:16:17,534 [64] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
gunicorn-registry stdout | 2025-02-07 03:16:17,535 [64] [DEBUG] [botocore.hooks] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
gunicorn-registry stdout | 2025-02-07 03:16:17,535 [64] [DEBUG] [botocore.hooks] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
gunicorn-registry stdout | 2025-02-07 03:16:17,536 [64] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
gunicorn-registry stdout | 2025-02-07 03:16:17,537 [64] [DEBUG] [botocore.hooks] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
gunicorn-registry stdout | 2025-02-07 03:16:17,537 [64] [DEBUG] [botocore.hooks] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
gunicorn-registry stdout | 2025-02-07 03:16:17,537 [64] [DEBUG] [botocore.hooks] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
gunicorn-registry stdout | 2025-02-07 03:16:17,628 [64] [DEBUG] [data.database] Configuring database
gunicorn-registry stdout | 2025-02-07 03:16:17,628 [64] [INFO] [data.database] Connection pooling enabled for postgresql; stale timeout: None; max connection count: None
gunicorn-registry stdout | 2025-02-07 03:16:17,629 [64] [INFO] [data.secscan_model] ===============================
gunicorn-registry stdout | 2025-02-07 03:16:17,629 [64] [INFO] [data.secscan_model] Using split secscan model: `[<data.secscan_model.secscan_v4_model.NoopV4SecurityScanner object at 0x7ff9bcde96a0>]`
gunicorn-registry stdout | 2025-02-07 03:16:17,629 [64] [INFO] [data.secscan_model] ===============================
gunicorn-registry stdout | 2025-02-07 03:16:17,629 [64] [DEBUG] [data.logs_model] Configuring log model
gunicorn-registry stdout | 2025-02-07 03:16:17,629 [64] [INFO] [data.logs_model] ===============================
gunicorn-registry stdout | 2025-02-07 03:16:17,629 [64] [INFO] [data.logs_model] Using logs model `<data.logs_model.table_logs_model.TableLogsModel object at 0x7ff9bcde9550>`
gunicorn-registry stdout | 2025-02-07 03:16:17,629 [64] [INFO] [data.logs_model] ===============================
gunicorn-registry stdout | 2025-02-07 03:16:18,024 [64] [DEBUG] [__config__] Starting registry gunicorn with 8 workers and gevent worker class
gunicorn-registry stderr | Traceback (most recent call last):
gunicorn-registry stderr |   File "src/gevent/_abstract_linkable.py", line 287, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
gunicorn-registry stderr |   File "src/gevent/_abstract_linkable.py", line 333, in gevent._gevent_c_abstract_linkable.AbstractLinkable._notify_links
gunicorn-registry stderr | AssertionError: (None, <callback at 0x7ff9bd8f8dc0 args=([],)>)
gunicorn-registry stderr | 2025-02-07T03:16:18Z 
gunicorn-registry stderr | <callback at 0x7ff9bd8f8dc0 args=([],)> failed with AssertionError
notificationworker stdout | 2025-02-07 03:16:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:18 UTC)" (scheduled at 2025-02-07 03:16:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:16:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:16:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 16, 18, 732941), True, datetime.datetime(2025, 2, 7, 3, 16, 18, 732941), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:16:18,734 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:28.732147+00:00 (in 9.998108 seconds)
notificationworker stdout | 2025-02-07 03:16:18,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:16:18,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:16:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:16:23,423 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:16:23,424 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:16:23 UTC)" (scheduled at 2025-02-07 03:16:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:16:23,424 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:16:23,424 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:16:23 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:16:23,424 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:03.419110+00:00 (in 39.994290 seconds)
gcworker stdout | 2025-02-07 03:16:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:16:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:16:24 UTC)" (scheduled at 2025-02-07 03:16:24.423055+00:00)
gcworker stdout | 2025-02-07 03:16:24,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 03:16:24,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:40.424250+00:00 (in 15.999812 seconds)
storagereplication stdout | 2025-02-07 03:16:24,427 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
gcworker stdout | 2025-02-07 03:16:24,430 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:16:24,430 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:16:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:16:54 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:16:26,345 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:16:27,345 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:16:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:38.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 03:16:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:38 UTC)" (scheduled at 2025-02-07 03:16:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:16:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:16:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 16, 28, 732882), True, datetime.datetime(2025, 2, 7, 3, 16, 28, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:16:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:16:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:16:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:16:29,238 [236] [DEBUG] [app] Starting request: urn:request:b7f3c875-31c5-4a47-9e9c-f126183e0fe9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:16:29,243 [236] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-registry stdout | 2025-02-07 03:16:29,250 [239] [DEBUG] [app] Starting request: urn:request:7d3a734f-23fc-434c-8f8e-fbd2b4ccd059 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:16:29,251 [239] [DEBUG] [app] Ending request: urn:request:7d3a734f-23fc-434c-8f8e-fbd2b4ccd059 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7d3a734f-23fc-434c-8f8e-fbd2b4ccd059', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.005 162 0.005)
gunicorn-registry stdout | 2025-02-07 03:16:29,252 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:29,252 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:29,253 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:16:29,257 [235] [DEBUG] [app] Starting request: urn:request:fd8e4fe7-521b-4125-95c7-5f0919e96cba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:16:29,258 [235] [DEBUG] [app] Ending request: urn:request:fd8e4fe7-521b-4125-95c7-5f0919e96cba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd8e4fe7-521b-4125-95c7-5f0919e96cba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.005 159 0.005)
gunicorn-web stdout | 2025-02-07 03:16:29,259 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:29,259 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:29,261 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 3, 16, 29, 260500), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 3, 16, 29, 260523), 'quay', None, datetime.datetime(2025, 2, 6, 3, 16, 29, 260535), datetime.datetime(2025, 1, 31, 3, 16, 29, 260544), None])
gunicorn-web stdout | 2025-02-07 03:16:29,269 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:16:29,269 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:16:29,273 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:16:29,273 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:16:29,274 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:16:29,275 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:16:29,276 [236] [DEBUG] [app] Ending request: urn:request:b7f3c875-31c5-4a47-9e9c-f126183e0fe9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b7f3c875-31c5-4a47-9e9c-f126183e0fe9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:16:29,277 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:16:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.043 118 0.042)
gunicorn-web stdout | 2025-02-07 03:16:29,277 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:16:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:16:29,279 [235] [DEBUG] [app] Starting request: urn:request:55bf7025-8350-4d00-98b6-f14a986f1577 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:16:29,283 [235] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-registry stdout | 2025-02-07 03:16:29,288 [239] [DEBUG] [app] Starting request: urn:request:893eb98f-e72a-4fbd-900f-84dd6b5779d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:16:29,289 [239] [DEBUG] [app] Ending request: urn:request:893eb98f-e72a-4fbd-900f-84dd6b5779d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:893eb98f-e72a-4fbd-900f-84dd6b5779d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:16:29,289 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:16:29,289 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:29,291 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:16:29,292 [235] [DEBUG] [app] Starting request: urn:request:1e2f615e-8545-4850-a67c-6e2fa7856018 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:16:29,292 [235] [DEBUG] [app] Ending request: urn:request:1e2f615e-8545-4850-a67c-6e2fa7856018 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e2f615e-8545-4850-a67c-6e2fa7856018', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:16:29,292 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:29,293 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:29,294 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 3, 16, 29, 293994), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 3, 16, 29, 294018), 'quay', None, datetime.datetime(2025, 2, 6, 3, 16, 29, 294031), datetime.datetime(2025, 1, 31, 3, 16, 29, 294037), None])
gunicorn-web stdout | 2025-02-07 03:16:29,302 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:16:29,303 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:16:29,307 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:16:29,307 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:16:29,308 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:16:29,309 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:16:29,310 [235] [DEBUG] [app] Ending request: urn:request:55bf7025-8350-4d00-98b6-f14a986f1577 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55bf7025-8350-4d00-98b6-f14a986f1577', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:16:29,310 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:16:29,311 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:16:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:16:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.033 118 0.033)
namespacegcworker stdout | 2025-02-07 03:16:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:16:31,622 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:16:31 UTC)" (scheduled at 2025-02-07 03:16:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:16:31,622 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:16:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 16, 31, 622166), True, datetime.datetime(2025, 2, 7, 3, 16, 31, 622166), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:16:31,623 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:32.625945+00:00 (in 1.002650 seconds)
namespacegcworker stdout | 2025-02-07 03:16:31,630 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:16:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:16:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:16:31,655 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:16:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:16:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:13.624301+00:00 (in 40.997881 seconds)
namespacegcworker stdout | 2025-02-07 03:16:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:32 UTC)" (scheduled at 2025-02-07 03:16:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:16:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:16:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:16:32,732 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:16:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:16:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:16:33 UTC)" (scheduled at 2025-02-07 03:16:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:16:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 16, 33, 920075), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:16:33,921 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:03.919286+00:00 (in 29.998261 seconds)
buildlogsarchiver stdout | 2025-02-07 03:16:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:16:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:16:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:03 UTC)" executed successfully
builder stdout | 2025-02-07 03:16:37,328 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:37,717 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:16:37 UTC)" (scheduled at 2025-02-07 03:16:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:37,718 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:37,718 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:56.713547+00:00 (in 18.995009 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:37,724 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:37,724 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:37,724 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:37 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:16:37,852 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:16:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:48.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 03:16:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:48 UTC)" (scheduled at 2025-02-07 03:16:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:16:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:16:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 16, 38, 732919), True, datetime.datetime(2025, 2, 7, 3, 16, 38, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:16:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:16:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:16:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:16:39,160 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:16:39,858 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:16:40,045 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:16:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:16:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:54.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 03:16:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:10 UTC)" (scheduled at 2025-02-07 03:16:40.424250+00:00)
gcworker stdout | 2025-02-07 03:16:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:16:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738897900431, None, 1, 0])
gcworker stdout | 2025-02-07 03:16:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:16:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:16:41,026 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:41,258 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:16:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:16:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:16:41 UTC)" (scheduled at 2025-02-07 03:16:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:16:41,833 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:11.831521+00:00 (in 29.998414 seconds)
autopruneworker stdout | 2025-02-07 03:16:41,838 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894601837, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:16:41,841 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:16:41,841 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:16:41,841 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:16:42,158 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:16:42,169 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:16:42,172 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:16:42,642 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:16:43,058 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:16:43,067 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:16:43,229 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:16:44,235 [236] [DEBUG] [app] Starting request: urn:request:eb9002c4-0968-4447-bd32-62269507fbfa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:16:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:16:44,239 [237] [DEBUG] [app] Starting request: urn:request:4be01509-0cff-48c9-acb0-4b75f3ceefe5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-registry stdout | 2025-02-07 03:16:44,241 [246] [DEBUG] [app] Starting request: urn:request:17653b23-7c79-4604-bff6-de90575727b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:16:44,242 [246] [DEBUG] [app] Ending request: urn:request:17653b23-7c79-4604-bff6-de90575727b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:17653b23-7c79-4604-bff6-de90575727b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.005 162 0.005)
gunicorn-registry stdout | 2025-02-07 03:16:44,243 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:44,244 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:16:44,245 [237] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-web stdout | 2025-02-07 03:16:44,246 [236] [DEBUG] [app] Starting request: urn:request:7bd6bd43-9376-4777-8d12-279664289cd0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:16:44,246 [236] [DEBUG] [app] Ending request: urn:request:7bd6bd43-9376-4777-8d12-279664289cd0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7bd6bd43-9376-4777-8d12-279664289cd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:16:44,246 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:44,247 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:44,247 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:16:44,247 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-registry stdout | 2025-02-07 03:16:44,251 [246] [DEBUG] [app] Starting request: urn:request:40cc5c65-77b1-457a-8c51-dec8acb3c620 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:16:44,251 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:16:44,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-registry stdout | 2025-02-07 03:16:44,251 [246] [DEBUG] [app] Ending request: urn:request:40cc5c65-77b1-457a-8c51-dec8acb3c620 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40cc5c65-77b1-457a-8c51-dec8acb3c620', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:16:44,252 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:16:44,252 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:44,254 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:16:44,256 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:16:44,257 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:16:44,258 [238] [DEBUG] [app] Starting request: urn:request:b5a1ab58-4a2e-4375-8eec-11f1cd90ca17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:16:44,258 [236] [DEBUG] [app] Ending request: urn:request:eb9002c4-0968-4447-bd32-62269507fbfa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb9002c4-0968-4447-bd32-62269507fbfa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:16:44,258 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:16:44,259 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:16:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:16:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gunicorn-web stdout | 2025-02-07 03:16:44,259 [238] [DEBUG] [app] Ending request: urn:request:b5a1ab58-4a2e-4375-8eec-11f1cd90ca17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b5a1ab58-4a2e-4375-8eec-11f1cd90ca17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.004 159 0.005)
gunicorn-web stdout | 2025-02-07 03:16:44,260 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:44,260 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:44,262 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 3, 16, 44, 261347), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 3, 16, 44, 261371), 'quay', None, datetime.datetime(2025, 2, 6, 3, 16, 44, 261383), datetime.datetime(2025, 1, 31, 3, 16, 44, 261393), None])
gunicorn-web stdout | 2025-02-07 03:16:44,269 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:16:44,270 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:16:44,273 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:16:44,273 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:16:44,275 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:16:44,276 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:16:44,277 [237] [DEBUG] [app] Ending request: urn:request:4be01509-0cff-48c9-acb0-4b75f3ceefe5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4be01509-0cff-48c9-acb0-4b75f3ceefe5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:16:44,278 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:16:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.043 118 0.043)
gunicorn-web stdout | 2025-02-07 03:16:44,278 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:16:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:16:44,729 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:16:45,143 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:16:45,433 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-web stdout | 2025-02-07 03:16:45,543 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:16:45,545 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:16:45,547 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:16:45,549 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:16:45,550 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 03:16:46,580 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:16:46,582 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:16:46,584 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:16:47,469 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:16:47,471 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:16:47,473 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:16:47,475 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:16:47,478 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:16:47,480 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:16:47,483 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:16:47,486 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:16:47,488 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:16:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:50.814677+00:00 (in 2.082084 seconds)
notificationworker stdout | 2025-02-07 03:16:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:58 UTC)" (scheduled at 2025-02-07 03:16:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:16:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:16:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 16, 48, 732877), True, datetime.datetime(2025, 2, 7, 3, 16, 48, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:16:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:16:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:16:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:16:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:16:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:16:58.732147+00:00 (in 7.917001 seconds)
notificationworker stdout | 2025-02-07 03:16:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:50 UTC)" (scheduled at 2025-02-07 03:16:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:16:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:16:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:16:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:16:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:10.424250+00:00 (in 16.000599 seconds)
gcworker stdout | 2025-02-07 03:16:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:24 UTC)" (scheduled at 2025-02-07 03:16:54.423055+00:00)
gcworker stdout | 2025-02-07 03:16:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:16:54,424 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:16:54,434 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:16:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:16:56,120 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:16:56 UTC)" (scheduled at 2025-02-07 03:16:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:16:56,121 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:56.118855+00:00 (in 59.997712 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:16:56,121 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:16:56,127 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:16:56,127 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:16:56,381 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:37.715878+00:00 (in 41.001863 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:56 UTC)" (scheduled at 2025-02-07 03:16:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:16:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:17:56 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:16:57,380 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:16:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:16:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:08.732147+00:00 (in 9.999527 seconds)
notificationworker stdout | 2025-02-07 03:16:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:08 UTC)" (scheduled at 2025-02-07 03:16:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:16:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:16:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 16, 58, 732894), True, datetime.datetime(2025, 2, 7, 3, 16, 58, 732894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:16:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:16:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:16:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:16:59,235 [238] [DEBUG] [app] Starting request: urn:request:f80e39ee-e3ad-42a2-a31f-d7d68e0bd0df (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:16:59,235 [235] [DEBUG] [app] Starting request: urn:request:6f26195a-132b-4da9-8331-de84d1495c3b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:16:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:16:59,238 [239] [DEBUG] [app] Starting request: urn:request:b3dc6b3d-132b-4e60-a67a-ddb0d670da96 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:16:59,239 [239] [DEBUG] [app] Ending request: urn:request:b3dc6b3d-132b-4e60-a67a-ddb0d670da96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b3dc6b3d-132b-4e60-a67a-ddb0d670da96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:16:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:59,240 [238] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-web stdout | 2025-02-07 03:16:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:16:59,241 [239] [DEBUG] [app] Starting request: urn:request:e5e1cb0d-5405-4604-9d5e-943c3b935b42 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:16:59,242 [239] [DEBUG] [app] Ending request: urn:request:e5e1cb0d-5405-4604-9d5e-943c3b935b42 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5e1cb0d-5405-4604-9d5e-943c3b935b42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:16:59,242 [235] [DEBUG] [app] Starting request: urn:request:7ce62246-428c-4dc3-bdbc-4d8121d2ce67 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:16:59,242 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:59,242 [235] [DEBUG] [app] Ending request: urn:request:7ce62246-428c-4dc3-bdbc-4d8121d2ce67 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ce62246-428c-4dc3-bdbc-4d8121d2ce67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:16:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:16:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:16:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:16:59,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:16:59,245 [238] [DEBUG] [app] Starting request: urn:request:b5f807f4-a0e2-4999-853b-b8dff4e8e299 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:16:59,245 [238] [DEBUG] [app] Ending request: urn:request:b5f807f4-a0e2-4999-853b-b8dff4e8e299 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b5f807f4-a0e2-4999-853b-b8dff4e8e299', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:16:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:16:59,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:16:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:16:59,246 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:16:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:16:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:16:59,248 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 3, 16, 59, 246886), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 3, 16, 59, 246908), 'quay', None, datetime.datetime(2025, 2, 6, 3, 16, 59, 246922), datetime.datetime(2025, 1, 31, 3, 16, 59, 246931), None])
gunicorn-web stdout | 2025-02-07 03:16:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:16:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:16:59,254 [235] [DEBUG] [app] Ending request: urn:request:6f26195a-132b-4da9-8331-de84d1495c3b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f26195a-132b-4da9-8331-de84d1495c3b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:16:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:16:59,255 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:16:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:16:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:16:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:16:59,255 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:16:59,259 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:16:59,259 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:16:59,260 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:16:59,261 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:16:59,262 [238] [DEBUG] [app] Ending request: urn:request:f80e39ee-e3ad-42a2-a31f-d7d68e0bd0df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f80e39ee-e3ad-42a2-a31f-d7d68e0bd0df', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:16:59,262 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:16:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.029 118 0.029)
gunicorn-web stdout | 2025-02-07 03:16:59,263 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:16:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:16:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:16:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:16:59 UTC)" (scheduled at 2025-02-07 03:16:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:16:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:16:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:16:59 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:16:59,939 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:04.936688+00:00 (in 4.997628 seconds)
teamsyncworker stdout | 2025-02-07 03:17:01,691 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:17:02,768 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:17:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:17:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:23.420886+00:00 (in 20.001299 seconds)
repositorygcworker stdout | 2025-02-07 03:17:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:03 UTC)" (scheduled at 2025-02-07 03:17:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:17:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:17:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 3, 419909), True, datetime.datetime(2025, 2, 7, 3, 17, 3, 419909), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:17:03,428 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:17:03,428 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:17:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:17:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:17:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:33.919286+00:00 (in 29.999511 seconds)
buildlogsarchiver stdout | 2025-02-07 03:17:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:33 UTC)" (scheduled at 2025-02-07 03:17:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:17:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 17, 3, 920077), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:17:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:17:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:17:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:17:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:17:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:59.937899+00:00 (in 55.000751 seconds)
exportactionlogsworker stdout | 2025-02-07 03:17:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:04 UTC)" (scheduled at 2025-02-07 03:17:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:17:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:17:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 4, 937457), True, datetime.datetime(2025, 2, 7, 3, 17, 4, 937457), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:17:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:17:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:17:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:17:07,358 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:17:07,888 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:17:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:18.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 03:17:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:18 UTC)" (scheduled at 2025-02-07 03:17:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:17:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:17:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 8, 733028), True, datetime.datetime(2025, 2, 7, 3, 17, 8, 733028), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:17:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:17:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:17:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:17:09,196 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:17:09,893 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:17:10,069 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:17:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:17:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:24.423055+00:00 (in 13.998292 seconds)
gcworker stdout | 2025-02-07 03:17:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:40 UTC)" (scheduled at 2025-02-07 03:17:10.424250+00:00)
gcworker stdout | 2025-02-07 03:17:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:17:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738897930431, None, 1, 0])
gcworker stdout | 2025-02-07 03:17:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:17:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:17:11,054 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:11,274 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:17:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:17:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:41.831521+00:00 (in 29.999282 seconds)
autopruneworker stdout | 2025-02-07 03:17:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:41 UTC)" (scheduled at 2025-02-07 03:17:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:17:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894631836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:17:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:17:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:17:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:17:12,171 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:17:12,205 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:17:12,208 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:17:12,678 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:17:13,080 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:17:13,092 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:17:13,241 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
namespacegcworker stdout | 2025-02-07 03:17:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:17:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:31.621131+00:00 (in 17.996339 seconds)
namespacegcworker stdout | 2025-02-07 03:17:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:22:13 UTC)" (scheduled at 2025-02-07 03:17:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:17:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 17, 13, 625085), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:17:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 17, 13, 625085), True, datetime.datetime(2025, 2, 7, 3, 17, 13, 625085), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:17:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 17, 13, 625085), True, datetime.datetime(2025, 2, 7, 3, 17, 13, 625085), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 17, 13, 625085), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:17:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:17:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:22:13 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:17:14,235 [236] [DEBUG] [app] Starting request: urn:request:aca170ea-9ae8-4ea1-bc0e-44029cd0f419 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:14,235 [237] [DEBUG] [app] Starting request: urn:request:ed3e4317-e9f9-4dce-8496-dbf0902ae9f1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:17:14,238 [239] [DEBUG] [app] Starting request: urn:request:41759321-5330-4fe7-889e-9219ba5b0296 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:14,238 [246] [DEBUG] [app] Starting request: urn:request:bd314575-2403-42c2-9a8f-c2a9cf735a6e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:14,239 [239] [DEBUG] [app] Ending request: urn:request:41759321-5330-4fe7-889e-9219ba5b0296 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:41759321-5330-4fe7-889e-9219ba5b0296', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:17:14,239 [246] [DEBUG] [app] Ending request: urn:request:bd314575-2403-42c2-9a8f-c2a9cf735a6e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd314575-2403-42c2-9a8f-c2a9cf735a6e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:17:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:17:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:17:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:14,242 [235] [DEBUG] [app] Starting request: urn:request:a29b0787-aa68-48d0-8d2b-986173cf5938 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:14,242 [235] [DEBUG] [app] Ending request: urn:request:a29b0787-aa68-48d0-8d2b-986173cf5938 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a29b0787-aa68-48d0-8d2b-986173cf5938', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:17:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:14,242 [238] [DEBUG] [app] Starting request: urn:request:ee3580b5-1cb2-4b8e-872d-f69cbb27b09c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:14,243 [238] [DEBUG] [app] Ending request: urn:request:ee3580b5-1cb2-4b8e-872d-f69cbb27b09c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee3580b5-1cb2-4b8e-872d-f69cbb27b09c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:17:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:17:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:14,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:14,253 [237] [DEBUG] [app] Ending request: urn:request:ed3e4317-e9f9-4dce-8496-dbf0902ae9f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed3e4317-e9f9-4dce-8496-dbf0902ae9f1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:17:14,254 [236] [DEBUG] [app] Ending request: urn:request:aca170ea-9ae8-4ea1-bc0e-44029cd0f419 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aca170ea-9ae8-4ea1-bc0e-44029cd0f419', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:17:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:17:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:17:14,740 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:17:15,179 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:17:15,459 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-web stdout | 2025-02-07 03:17:15,551 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:17:15,554 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:17:15,557 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:17:15,559 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:17:15,561 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 03:17:16,589 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:17:16,592 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:17:16,594 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:17:17,477 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:17:17,480 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:17:17,483 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:17:17,487 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:17:17,490 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:17:17,492 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:17:17,495 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:17:17,498 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:17:17,501 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:17:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:28.732147+00:00 (in 9.999503 seconds)
notificationworker stdout | 2025-02-07 03:17:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:28 UTC)" (scheduled at 2025-02-07 03:17:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:17:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:17:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 18, 732964), True, datetime.datetime(2025, 2, 7, 3, 17, 18, 732964), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:17:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:17:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:17:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:17:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:17:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:03.419110+00:00 (in 39.997761 seconds)
repositorygcworker stdout | 2025-02-07 03:17:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:23 UTC)" (scheduled at 2025-02-07 03:17:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:17:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:17:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:17:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:17:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:40.424250+00:00 (in 16.000698 seconds)
gcworker stdout | 2025-02-07 03:17:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:54 UTC)" (scheduled at 2025-02-07 03:17:24.423055+00:00)
gcworker stdout | 2025-02-07 03:17:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:17:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:17:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:17:24,465 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:17:26,407 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:17:27,414 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:17:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:33.814166+00:00 (in 5.081511 seconds)
notificationworker stdout | 2025-02-07 03:17:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:38 UTC)" (scheduled at 2025-02-07 03:17:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:17:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:17:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 28, 732970), True, datetime.datetime(2025, 2, 7, 3, 17, 28, 732970), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:17:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:17:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:17:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:17:29,236 [236] [DEBUG] [app] Starting request: urn:request:4d4e3bf6-c8c5-496e-ad87-66b50c5e0f4f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:29,236 [237] [DEBUG] [app] Starting request: urn:request:4461e879-f647-467b-bf7b-5ec4d1fb2cf1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:17:29,239 [239] [DEBUG] [app] Starting request: urn:request:a0402743-1675-49fc-85d0-a05c50b10a7e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:29,239 [239] [DEBUG] [app] Ending request: urn:request:a0402743-1675-49fc-85d0-a05c50b10a7e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0402743-1675-49fc-85d0-a05c50b10a7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:17:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:17:29,240 [243] [DEBUG] [app] Starting request: urn:request:37e0386d-981f-49f5-bb29-ef50db4ce6c5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:29,241 [243] [DEBUG] [app] Ending request: urn:request:37e0386d-981f-49f5-bb29-ef50db4ce6c5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:37e0386d-981f-49f5-bb29-ef50db4ce6c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.004 162 0.004)
gunicorn-registry stdout | 2025-02-07 03:17:29,242 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:29,243 [237] [DEBUG] [app] Starting request: urn:request:51c3804e-8088-4b24-bfcd-d1bb102e9c3f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:29,243 [237] [DEBUG] [app] Ending request: urn:request:51c3804e-8088-4b24-bfcd-d1bb102e9c3f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:51c3804e-8088-4b24-bfcd-d1bb102e9c3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:17:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:29,244 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:17:29,245 [235] [DEBUG] [app] Starting request: urn:request:fef2552b-7adb-4a6d-97ee-68e65aca476f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:29,246 [235] [DEBUG] [app] Ending request: urn:request:fef2552b-7adb-4a6d-97ee-68e65aca476f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fef2552b-7adb-4a6d-97ee-68e65aca476f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:29,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:17:29,246 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:29,247 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:29,247 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:17:29,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:29,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:29,251 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:29,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:29,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:29,255 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:29,256 [237] [DEBUG] [app] Ending request: urn:request:4461e879-f647-467b-bf7b-5ec4d1fb2cf1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4461e879-f647-467b-bf7b-5ec4d1fb2cf1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:17:29,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:17:29,256 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:29,258 [236] [DEBUG] [app] Ending request: urn:request:4d4e3bf6-c8c5-496e-ad87-66b50c5e0f4f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d4e3bf6-c8c5-496e-ad87-66b50c5e0f4f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:17:29,258 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:29,258 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
namespacegcworker stdout | 2025-02-07 03:17:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:17:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:32.625945+00:00 (in 1.004279 seconds)
namespacegcworker stdout | 2025-02-07 03:17:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:31 UTC)" (scheduled at 2025-02-07 03:17:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:17:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:17:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 31, 621964), True, datetime.datetime(2025, 2, 7, 3, 17, 31, 621964), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:17:31,630 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:17:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:17:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:17:31,727 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:17:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:17:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:31.621131+00:00 (in 58.994730 seconds)
namespacegcworker stdout | 2025-02-07 03:17:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:32 UTC)" (scheduled at 2025-02-07 03:17:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:17:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:17:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:17:32,805 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 03:17:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:38.732147+00:00 (in 4.917460 seconds)
notificationworker stdout | 2025-02-07 03:17:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:22:33 UTC)" (scheduled at 2025-02-07 03:17:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:17:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 17, 33, 814972), 'notification/%'])
notificationworker stdout | 2025-02-07 03:17:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 17, 33, 814972), True, datetime.datetime(2025, 2, 7, 3, 17, 33, 814972), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:17:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 17, 33, 814972), True, datetime.datetime(2025, 2, 7, 3, 17, 33, 814972), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 17, 33, 814972), 'notification/%'])
notificationworker stdout | 2025-02-07 03:17:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:17:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:22:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:17:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:17:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:03.919286+00:00 (in 29.999499 seconds)
buildlogsarchiver stdout | 2025-02-07 03:17:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:03 UTC)" (scheduled at 2025-02-07 03:17:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:17:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 17, 33, 920074), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:17:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:17:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:17:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:03 UTC)" executed successfully
builder stdout | 2025-02-07 03:17:37,388 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:56.713547+00:00 (in 18.997210 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:37 UTC)" (scheduled at 2025-02-07 03:17:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:37 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:17:37,901 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:17:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:48.732147+00:00 (in 9.999482 seconds)
notificationworker stdout | 2025-02-07 03:17:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:48 UTC)" (scheduled at 2025-02-07 03:17:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:17:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:17:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 38, 732975), True, datetime.datetime(2025, 2, 7, 3, 17, 38, 732975), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:17:38,744 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:17:38,744 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:17:38,744 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:17:39,233 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:17:39,929 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:17:40,090 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:17:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:17:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:54.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 03:17:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:10 UTC)" (scheduled at 2025-02-07 03:17:40.424250+00:00)
gcworker stdout | 2025-02-07 03:17:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:17:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738897960431, None, 1, 0])
gcworker stdout | 2025-02-07 03:17:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:17:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:17:41,075 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:41,309 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:17:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:17:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:11.831521+00:00 (in 29.999510 seconds)
autopruneworker stdout | 2025-02-07 03:17:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:11 UTC)" (scheduled at 2025-02-07 03:17:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:17:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894661836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:17:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:17:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:17:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:17:42,178 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:17:42,234 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:17:42,237 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:17:42,715 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:17:43,106 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:17:43,127 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:17:43,272 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:17:44,235 [237] [DEBUG] [app] Starting request: urn:request:a53fa17c-8161-428b-9b8f-bab806850c33 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:44,235 [235] [DEBUG] [app] Starting request: urn:request:2f10c2cf-94d4-4125-acff-5cc811d82ffb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:17:44,238 [246] [DEBUG] [app] Starting request: urn:request:e194aa05-1d46-40a9-b50a-9d8bd2cec443 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:44,238 [239] [DEBUG] [app] Starting request: urn:request:4d0dce10-ede3-4052-8d2a-9e5cbfb15c6e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:44,239 [239] [DEBUG] [app] Ending request: urn:request:4d0dce10-ede3-4052-8d2a-9e5cbfb15c6e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d0dce10-ede3-4052-8d2a-9e5cbfb15c6e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:17:44,239 [246] [DEBUG] [app] Ending request: urn:request:e194aa05-1d46-40a9-b50a-9d8bd2cec443 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e194aa05-1d46-40a9-b50a-9d8bd2cec443', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:17:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:17:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:17:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:44,242 [235] [DEBUG] [app] Starting request: urn:request:bbe4447b-31e5-4e3b-b967-90d8b85af91d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:44,243 [235] [DEBUG] [app] Ending request: urn:request:bbe4447b-31e5-4e3b-b967-90d8b85af91d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bbe4447b-31e5-4e3b-b967-90d8b85af91d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:44,243 [236] [DEBUG] [app] Starting request: urn:request:43738d2f-0b39-4bcd-ad0f-7fa1908d3f31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:17:44,243 [236] [DEBUG] [app] Ending request: urn:request:43738d2f-0b39-4bcd-ad0f-7fa1908d3f31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:43738d2f-0b39-4bcd-ad0f-7fa1908d3f31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:17:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:17:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:17:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:44,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:44,256 [235] [DEBUG] [app] Ending request: urn:request:2f10c2cf-94d4-4125-acff-5cc811d82ffb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f10c2cf-94d4-4125-acff-5cc811d82ffb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:17:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:17:44,256 [237] [DEBUG] [app] Ending request: urn:request:a53fa17c-8161-428b-9b8f-bab806850c33 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a53fa17c-8161-428b-9b8f-bab806850c33', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:17:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:44,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
namespacegcworker stdout | 2025-02-07 03:17:44,776 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:17:45,216 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:17:45,493 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-web stdout | 2025-02-07 03:17:45,561 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:17:45,565 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:17:45,568 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:17:45,571 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:17:45,574 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 03:17:46,599 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:17:46,602 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:17:46,604 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:17:47,488 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:17:47,491 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:17:47,496 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:17:47,499 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:17:47,502 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:17:47,505 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:17:47,508 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:17:47,511 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:17:47,514 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:17:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:50.814677+00:00 (in 2.082016 seconds)
notificationworker stdout | 2025-02-07 03:17:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:58 UTC)" (scheduled at 2025-02-07 03:17:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:17:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:17:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 48, 732968), True, datetime.datetime(2025, 2, 7, 3, 17, 48, 732968), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:17:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:17:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:17:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:17:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:17:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:17:58.732147+00:00 (in 7.916945 seconds)
notificationworker stdout | 2025-02-07 03:17:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:50 UTC)" (scheduled at 2025-02-07 03:17:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:17:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:17:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:17:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:17:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:10.424250+00:00 (in 16.000712 seconds)
gcworker stdout | 2025-02-07 03:17:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:24 UTC)" (scheduled at 2025-02-07 03:17:54.423055+00:00)
gcworker stdout | 2025-02-07 03:17:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:17:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:17:54,487 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:17:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:17:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:56.118855+00:00 (in 59.999005 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:17:56,120 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:56 UTC)" (scheduled at 2025-02-07 03:17:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:17:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:17:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:17:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:17:56,437 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:37.715878+00:00 (in 41.001859 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:56 UTC)" (scheduled at 2025-02-07 03:17:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:17:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:56 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:17:57,439 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:17:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:17:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:08.732147+00:00 (in 9.999459 seconds)
notificationworker stdout | 2025-02-07 03:17:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:08 UTC)" (scheduled at 2025-02-07 03:17:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:17:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:17:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 17, 58, 732993), True, datetime.datetime(2025, 2, 7, 3, 17, 58, 732993), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:17:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:17:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:17:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:17:59,235 [236] [DEBUG] [app] Starting request: urn:request:e9635680-58dc-4d7d-80cc-e751eb6f71ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:59,235 [235] [DEBUG] [app] Starting request: urn:request:e00cf3cc-b5e4-4ce4-bebd-39122c880b8c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:17:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:59,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:17:59,239 [239] [DEBUG] [app] Starting request: urn:request:d32c1176-a66a-4afb-9064-356c27f89d41 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:59,239 [239] [DEBUG] [app] Ending request: urn:request:d32c1176-a66a-4afb-9064-356c27f89d41 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d32c1176-a66a-4afb-9064-356c27f89d41', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:17:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:17:59,240 [244] [DEBUG] [app] Starting request: urn:request:f13d332e-1e16-4ad3-af94-1f65f98567e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:17:59,241 [244] [DEBUG] [app] Ending request: urn:request:f13d332e-1e16-4ad3-af94-1f65f98567e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f13d332e-1e16-4ad3-af94-1f65f98567e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.004 162 0.004)
gunicorn-registry stdout | 2025-02-07 03:17:59,242 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:59,243 [238] [DEBUG] [app] Starting request: urn:request:0d399ddd-12ae-4f90-a1e3-a0bc46282d78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:59,243 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:17:59,243 [238] [DEBUG] [app] Ending request: urn:request:0d399ddd-12ae-4f90-a1e3-a0bc46282d78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d399ddd-12ae-4f90-a1e3-a0bc46282d78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:17:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:17:59,244 [238] [DEBUG] [app] Starting request: urn:request:47967998-769c-47cf-bb91-30d65f152278 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:17:59,244 [238] [DEBUG] [app] Ending request: urn:request:47967998-769c-47cf-bb91-30d65f152278 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47967998-769c-47cf-bb91-30d65f152278', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:17:59,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:17:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:17:59,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:17:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:17:59,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:59,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:17:59,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:17:59,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:17:59,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:59,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:17:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:17:59,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:17:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:59,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:17:59,256 [236] [DEBUG] [app] Ending request: urn:request:e9635680-58dc-4d7d-80cc-e751eb6f71ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e9635680-58dc-4d7d-80cc-e751eb6f71ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:17:59,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:17:59,256 [235] [DEBUG] [app] Ending request: urn:request:e00cf3cc-b5e4-4ce4-bebd-39122c880b8c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e00cf3cc-b5e4-4ce4-bebd-39122c880b8c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:17:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:17:59,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:17:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:17:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
exportactionlogsworker stdout | 2025-02-07 03:17:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:17:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:04.936688+00:00 (in 4.998306 seconds)
exportactionlogsworker stdout | 2025-02-07 03:17:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:59 UTC)" (scheduled at 2025-02-07 03:17:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:17:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:17:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:18:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:18:01,763 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:18:02,831 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:18:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:18:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:23.420886+00:00 (in 20.001332 seconds)
repositorygcworker stdout | 2025-02-07 03:18:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:03 UTC)" (scheduled at 2025-02-07 03:18:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:18:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:18:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 3, 419870), True, datetime.datetime(2025, 2, 7, 3, 18, 3, 419870), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:18:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:18:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:18:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:18:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:18:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:33.919286+00:00 (in 29.999485 seconds)
buildlogsarchiver stdout | 2025-02-07 03:18:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:33 UTC)" (scheduled at 2025-02-07 03:18:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:18:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 18, 3, 920091), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:18:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:18:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:18:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:18:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:18:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:59.937899+00:00 (in 55.000772 seconds)
exportactionlogsworker stdout | 2025-02-07 03:18:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:04 UTC)" (scheduled at 2025-02-07 03:18:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:18:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:18:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 4, 937420), True, datetime.datetime(2025, 2, 7, 3, 18, 4, 937420), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:18:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:18:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:18:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:18:07,406 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:18:07,936 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:18:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:18:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:18.732147+00:00 (in 9.999484 seconds)
notificationworker stdout | 2025-02-07 03:18:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:18 UTC)" (scheduled at 2025-02-07 03:18:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:18:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:18:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 8, 732960), True, datetime.datetime(2025, 2, 7, 3, 18, 8, 732960), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:18:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:18:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:18:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:18:09,241 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:18:09,942 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:18:10,106 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:18:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:18:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:24.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 03:18:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:40 UTC)" (scheduled at 2025-02-07 03:18:10.424250+00:00)
gcworker stdout | 2025-02-07 03:18:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:18:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738897990431, None, 1, 0])
gcworker stdout | 2025-02-07 03:18:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:18:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:18:11,088 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:11,346 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:18:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:18:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:41.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 03:18:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:41 UTC)" (scheduled at 2025-02-07 03:18:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:18:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894691836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:18:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:18:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:18:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:18:12,194 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:18:12,271 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:18:12,273 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:18:12,730 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:18:13,140 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:18:13,163 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:18:13,297 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:18:14,236 [235] [DEBUG] [app] Starting request: urn:request:b884004f-69ee-4a81-b278-0e6da7e2d1ee (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:14,236 [236] [DEBUG] [app] Starting request: urn:request:eda81c7b-ef23-446a-82ec-f682415c0c9b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:18:14,239 [246] [DEBUG] [app] Starting request: urn:request:7c0ecc36-b2ab-4efc-92ac-af3ae0d002f7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:18:14,239 [246] [DEBUG] [app] Ending request: urn:request:7c0ecc36-b2ab-4efc-92ac-af3ae0d002f7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7c0ecc36-b2ab-4efc-92ac-af3ae0d002f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:18:14,239 [239] [DEBUG] [app] Starting request: urn:request:91ec5db1-82f0-4945-a279-f62eb11c9293 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:18:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:18:14,239 [239] [DEBUG] [app] Ending request: urn:request:91ec5db1-82f0-4945-a279-f62eb11c9293 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:91ec5db1-82f0-4945-a279-f62eb11c9293', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:18:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:18:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:14,242 [236] [DEBUG] [app] Starting request: urn:request:494c1089-bd29-4d80-afee-6c3f0415e8d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:14,242 [236] [DEBUG] [app] Ending request: urn:request:494c1089-bd29-4d80-afee-6c3f0415e8d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:494c1089-bd29-4d80-afee-6c3f0415e8d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:18:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:14,243 [238] [DEBUG] [app] Starting request: urn:request:d9102d85-0be3-479c-a64b-2039116ea99d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:14,244 [238] [DEBUG] [app] Ending request: urn:request:d9102d85-0be3-479c-a64b-2039116ea99d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9102d85-0be3-479c-a64b-2039116ea99d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:14,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:18:14,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:18:14,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:14,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:14,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:14,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:18:14,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:14,254 [235] [DEBUG] [app] Ending request: urn:request:b884004f-69ee-4a81-b278-0e6da7e2d1ee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b884004f-69ee-4a81-b278-0e6da7e2d1ee', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:18:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:18:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:18:14,255 [236] [DEBUG] [app] Ending request: urn:request:eda81c7b-ef23-446a-82ec-f682415c0c9b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eda81c7b-ef23-446a-82ec-f682415c0c9b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:18:14,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:18:14,813 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:18:15,252 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:18:15,529 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-web stdout | 2025-02-07 03:18:15,568 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:18:15,571 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:18:15,575 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:18:15,577 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:18:15,581 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 03:18:16,607 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:18:16,609 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:18:16,612 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:18:17,498 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:18:17,501 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:18:17,505 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:18:17,509 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:18:17,512 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:18:17,515 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:18:17,520 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:18:17,523 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:18:17,526 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:18:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:18:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:28.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 03:18:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:28 UTC)" (scheduled at 2025-02-07 03:18:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:18:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:18:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 18, 732896), True, datetime.datetime(2025, 2, 7, 3, 18, 18, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:18:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:18:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:18:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:18:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:18:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:32.420707+00:00 (in 8.999367 seconds)
repositorygcworker stdout | 2025-02-07 03:18:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:23 UTC)" (scheduled at 2025-02-07 03:18:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:18:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:18:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:18:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:18:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 03:18:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:54 UTC)" (scheduled at 2025-02-07 03:18:24.423055+00:00)
gcworker stdout | 2025-02-07 03:18:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:18:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:18:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:18:24,511 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:18:26,473 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:18:27,474 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:18:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:18:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:38.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 03:18:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:38 UTC)" (scheduled at 2025-02-07 03:18:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:18:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:18:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 28, 732861), True, datetime.datetime(2025, 2, 7, 3, 18, 28, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:18:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:18:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:18:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:18:29,235 [235] [DEBUG] [app] Starting request: urn:request:93497eb3-396b-46c9-a943-6b8cfe627dbb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:29,235 [237] [DEBUG] [app] Starting request: urn:request:f1211cb4-674a-4b60-86af-6bedc5f3fa8e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:18:29,238 [239] [DEBUG] [app] Starting request: urn:request:86a55f34-7c94-4fa7-b3a4-94d662b1e685 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:18:29,238 [239] [DEBUG] [app] Ending request: urn:request:86a55f34-7c94-4fa7-b3a4-94d662b1e685 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86a55f34-7c94-4fa7-b3a4-94d662b1e685', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:18:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:18:29,239 [246] [DEBUG] [app] Starting request: urn:request:9f046ad6-480b-4f4d-bd32-19231b70dff5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:18:29,239 [246] [DEBUG] [app] Ending request: urn:request:9f046ad6-480b-4f4d-bd32-19231b70dff5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9f046ad6-480b-4f4d-bd32-19231b70dff5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:18:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:18:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:18:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:29,241 [235] [DEBUG] [app] Starting request: urn:request:115bee86-0910-48bf-8880-193689404131 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:29,241 [235] [DEBUG] [app] Ending request: urn:request:115bee86-0910-48bf-8880-193689404131 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:115bee86-0910-48bf-8880-193689404131', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:18:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:29,242 [236] [DEBUG] [app] Starting request: urn:request:948e3ec9-40fd-4859-9da7-223e2ed76dac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:29,243 [236] [DEBUG] [app] Ending request: urn:request:948e3ec9-40fd-4859-9da7-223e2ed76dac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:948e3ec9-40fd-4859-9da7-223e2ed76dac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:18:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:18:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:18:29,254 [235] [DEBUG] [app] Ending request: urn:request:93497eb3-396b-46c9-a943-6b8cfe627dbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93497eb3-396b-46c9-a943-6b8cfe627dbb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:29,254 [237] [DEBUG] [app] Ending request: urn:request:f1211cb4-674a-4b60-86af-6bedc5f3fa8e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1211cb4-674a-4b60-86af-6bedc5f3fa8e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:18:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:18:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:18:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:18:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:18:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:32.625945+00:00 (in 1.004368 seconds)
namespacegcworker stdout | 2025-02-07 03:18:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:31 UTC)" (scheduled at 2025-02-07 03:18:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:18:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:18:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 31, 621891), True, datetime.datetime(2025, 2, 7, 3, 18, 31, 621891), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:18:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:18:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:18:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:18:31,799 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:18:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:18:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:03.419110+00:00 (in 30.997961 seconds)
repositorygcworker stdout | 2025-02-07 03:18:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:23:32 UTC)" (scheduled at 2025-02-07 03:18:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:18:32,422 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 18, 32, 421455), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:18:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 18, 32, 421455), True, datetime.datetime(2025, 2, 7, 3, 18, 32, 421455), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:18:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 18, 32, 421455), True, datetime.datetime(2025, 2, 7, 3, 18, 32, 421455), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 18, 32, 421455), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:18:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:18:32,432 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:23:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:18:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:18:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:31.621131+00:00 (in 58.994726 seconds)
namespacegcworker stdout | 2025-02-07 03:18:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:32 UTC)" (scheduled at 2025-02-07 03:18:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:18:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:18:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:18:32,848 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:18:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:18:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:03.919286+00:00 (in 29.999507 seconds)
buildlogsarchiver stdout | 2025-02-07 03:18:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:03 UTC)" (scheduled at 2025-02-07 03:18:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:18:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 18, 33, 920042), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:18:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:18:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:18:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:03 UTC)" executed successfully
builder stdout | 2025-02-07 03:18:37,434 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:56.713547+00:00 (in 18.997207 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:37 UTC)" (scheduled at 2025-02-07 03:18:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:37 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:18:37,947 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:18:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:18:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:48.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 03:18:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:48 UTC)" (scheduled at 2025-02-07 03:18:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:18:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:18:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 38, 732889), True, datetime.datetime(2025, 2, 7, 3, 18, 38, 732889), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:18:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:18:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:18:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:18:39,274 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:18:39,976 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:18:40,136 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:18:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:18:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:54.423055+00:00 (in 13.998267 seconds)
gcworker stdout | 2025-02-07 03:18:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:10 UTC)" (scheduled at 2025-02-07 03:18:40.424250+00:00)
gcworker stdout | 2025-02-07 03:18:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:18:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898020431, None, 1, 0])
gcworker stdout | 2025-02-07 03:18:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:18:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:18:41,124 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:41,382 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:18:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:18:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:11.831521+00:00 (in 29.999532 seconds)
autopruneworker stdout | 2025-02-07 03:18:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:11 UTC)" (scheduled at 2025-02-07 03:18:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:18:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894721836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:18:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:18:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:18:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:18:42,207 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:18:42,307 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:18:42,310 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:18:42,748 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:18:43,170 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:18:43,199 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:18:43,333 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:18:44,236 [237] [DEBUG] [app] Starting request: urn:request:ea899ca1-3794-49ac-980f-3b2211807592 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:44,236 [238] [DEBUG] [app] Starting request: urn:request:60cf91a9-82c4-49d1-95fb-f6c42ea99f92 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:18:44,239 [246] [DEBUG] [app] Starting request: urn:request:11f6b149-7574-4900-8aaa-ae7926a987b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:18:44,239 [246] [DEBUG] [app] Ending request: urn:request:11f6b149-7574-4900-8aaa-ae7926a987b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11f6b149-7574-4900-8aaa-ae7926a987b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:18:44,239 [239] [DEBUG] [app] Starting request: urn:request:562f5e38-fd46-413a-8bb1-3691b1cdc37f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:18:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:18:44,240 [239] [DEBUG] [app] Ending request: urn:request:562f5e38-fd46-413a-8bb1-3691b1cdc37f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:562f5e38-fd46-413a-8bb1-3691b1cdc37f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:18:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:18:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:44,242 [237] [DEBUG] [app] Starting request: urn:request:73116255-ae98-4605-a712-a76e11d19a4f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:44,242 [237] [DEBUG] [app] Ending request: urn:request:73116255-ae98-4605-a712-a76e11d19a4f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73116255-ae98-4605-a712-a76e11d19a4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:18:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:18:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:44,243 [236] [DEBUG] [app] Starting request: urn:request:2cbeb2fd-f982-41a6-a989-b5a90aaf9fa0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:44,244 [236] [DEBUG] [app] Ending request: urn:request:2cbeb2fd-f982-41a6-a989-b5a90aaf9fa0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2cbeb2fd-f982-41a6-a989-b5a90aaf9fa0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:18:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:18:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:18:44,255 [237] [DEBUG] [app] Ending request: urn:request:ea899ca1-3794-49ac-980f-3b2211807592 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ea899ca1-3794-49ac-980f-3b2211807592', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:18:44,255 [238] [DEBUG] [app] Ending request: urn:request:60cf91a9-82c4-49d1-95fb-f6c42ea99f92 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:60cf91a9-82c4-49d1-95fb-f6c42ea99f92', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:18:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:18:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:18:44,849 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:18:45,279 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:18:45,565 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-web stdout | 2025-02-07 03:18:45,577 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:18:45,580 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:18:45,582 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:18:45,585 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:18:45,588 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 03:18:46,614 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:18:46,618 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:18:46,620 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:18:47,510 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:18:47,514 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:18:47,517 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:18:47,521 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:18:47,523 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:18:47,526 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:18:47,529 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:18:47,532 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:18:47,535 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:18:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:18:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:50.814677+00:00 (in 2.082060 seconds)
notificationworker stdout | 2025-02-07 03:18:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:58 UTC)" (scheduled at 2025-02-07 03:18:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:18:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:18:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 48, 732929), True, datetime.datetime(2025, 2, 7, 3, 18, 48, 732929), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:18:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:18:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:18:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:18:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:18:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:18:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:18:58.732147+00:00 (in 7.916975 seconds)
notificationworker stdout | 2025-02-07 03:18:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:50 UTC)" (scheduled at 2025-02-07 03:18:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:18:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:18:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:18:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:18:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:10.424250+00:00 (in 16.000678 seconds)
gcworker stdout | 2025-02-07 03:18:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:24 UTC)" (scheduled at 2025-02-07 03:18:54.423055+00:00)
gcworker stdout | 2025-02-07 03:18:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:18:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:18:54,538 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:18:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:18:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:56.118855+00:00 (in 59.999525 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:18:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:56 UTC)" (scheduled at 2025-02-07 03:18:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:18:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:18:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:18:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:18:56,505 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:37.715878+00:00 (in 41.001893 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:56 UTC)" (scheduled at 2025-02-07 03:18:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:18:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:56 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:18:57,510 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:18:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:18:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:08.732147+00:00 (in 9.999517 seconds)
notificationworker stdout | 2025-02-07 03:18:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:08 UTC)" (scheduled at 2025-02-07 03:18:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:18:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:18:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 18, 58, 732830), True, datetime.datetime(2025, 2, 7, 3, 18, 58, 732830), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:18:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:18:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:18:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:18:59,235 [236] [DEBUG] [app] Starting request: urn:request:17eaed68-7bd0-4555-ae76-f83c05f5c0b9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:59,235 [238] [DEBUG] [app] Starting request: urn:request:f82492e1-7e42-4350-aeac-408c28d05cf6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:18:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:18:59,238 [239] [DEBUG] [app] Starting request: urn:request:54563707-5d6d-4099-ba1a-9a8328b0fd19 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:18:59,239 [239] [DEBUG] [app] Ending request: urn:request:54563707-5d6d-4099-ba1a-9a8328b0fd19 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54563707-5d6d-4099-ba1a-9a8328b0fd19', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:18:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:18:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:18:59,240 [241] [DEBUG] [app] Starting request: urn:request:85ee1938-ccea-4eb5-9362-f7b7da416910 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:18:59,240 [241] [DEBUG] [app] Ending request: urn:request:85ee1938-ccea-4eb5-9362-f7b7da416910 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85ee1938-ccea-4eb5-9362-f7b7da416910', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 162 0.003)
gunicorn-registry stdout | 2025-02-07 03:18:59,241 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:18:59,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:59,242 [235] [DEBUG] [app] Starting request: urn:request:fce77541-6698-41e3-a4a0-c85d50e3e99d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:59,242 [235] [DEBUG] [app] Ending request: urn:request:fce77541-6698-41e3-a4a0-c85d50e3e99d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fce77541-6698-41e3-a4a0-c85d50e3e99d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:18:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:18:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:59,243 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:18:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:59,245 [235] [DEBUG] [app] Starting request: urn:request:82532f11-4fd2-42f5-a0ff-369fe0c86fd1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:18:59,245 [235] [DEBUG] [app] Ending request: urn:request:82532f11-4fd2-42f5-a0ff-369fe0c86fd1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82532f11-4fd2-42f5-a0ff-369fe0c86fd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:18:59,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:18:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:18:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:18:59,246 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:18:59,246 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:18:59,246 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:18:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:59,250 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:18:59,250 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:18:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:59,254 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:18:59,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:18:59,256 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:18:59,256 [238] [DEBUG] [app] Ending request: urn:request:f82492e1-7e42-4350-aeac-408c28d05cf6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f82492e1-7e42-4350-aeac-408c28d05cf6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:59,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:18:59,257 [236] [DEBUG] [app] Ending request: urn:request:17eaed68-7bd0-4555-ae76-f83c05f5c0b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:17eaed68-7bd0-4555-ae76-f83c05f5c0b9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:18:59,257 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:18:59,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:18:59,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:18:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:18:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
exportactionlogsworker stdout | 2025-02-07 03:18:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:18:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:04.936688+00:00 (in 4.998305 seconds)
exportactionlogsworker stdout | 2025-02-07 03:18:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:59 UTC)" (scheduled at 2025-02-07 03:18:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:18:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:18:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:19:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:19:01,824 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:19:02,858 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:19:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:19:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:23.420886+00:00 (in 20.001277 seconds)
repositorygcworker stdout | 2025-02-07 03:19:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:03 UTC)" (scheduled at 2025-02-07 03:19:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:19:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:19:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 3, 419930), True, datetime.datetime(2025, 2, 7, 3, 19, 3, 419930), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:19:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:19:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:19:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:19:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:19:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:33.919286+00:00 (in 29.999513 seconds)
buildlogsarchiver stdout | 2025-02-07 03:19:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:33 UTC)" (scheduled at 2025-02-07 03:19:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:19:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 19, 3, 920056), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:19:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:19:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:19:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:19:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:19:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:52.937648+00:00 (in 48.000522 seconds)
exportactionlogsworker stdout | 2025-02-07 03:19:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:04 UTC)" (scheduled at 2025-02-07 03:19:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:19:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:19:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 4, 937436), True, datetime.datetime(2025, 2, 7, 3, 19, 4, 937436), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:19:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:19:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:19:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:19:07,471 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:19:07,973 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:19:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:19:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:18.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 03:19:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:18 UTC)" (scheduled at 2025-02-07 03:19:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:19:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:19:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 8, 732911), True, datetime.datetime(2025, 2, 7, 3, 19, 8, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:19:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:19:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:19:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:19:09,281 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:19:10,012 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:19:10,149 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:19:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:19:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:24.423055+00:00 (in 13.998255 seconds)
gcworker stdout | 2025-02-07 03:19:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:40 UTC)" (scheduled at 2025-02-07 03:19:10.424250+00:00)
gcworker stdout | 2025-02-07 03:19:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:19:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898050431, None, 1, 0])
gcworker stdout | 2025-02-07 03:19:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:19:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:19:11,156 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:11,411 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:19:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:19:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 03:19:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:41 UTC)" (scheduled at 2025-02-07 03:19:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:19:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894751836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:19:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:19:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:19:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:19:12,231 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:19:12,324 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:19:12,343 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:19:12,773 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:19:13,198 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:19:13,233 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:19:13,370 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:19:14,235 [235] [DEBUG] [app] Starting request: urn:request:b75a9080-1c81-4609-9a6b-41480ca3dde1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:14,235 [237] [DEBUG] [app] Starting request: urn:request:4030f5a0-8529-4b0e-b419-1033085dacf3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:19:14,238 [246] [DEBUG] [app] Starting request: urn:request:313bed9d-6e7b-40ce-b6a9-3275895beaf7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:19:14,238 [243] [DEBUG] [app] Starting request: urn:request:fb35931c-2ec7-4260-809b-dfc373e268fb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:19:14,239 [246] [DEBUG] [app] Ending request: urn:request:313bed9d-6e7b-40ce-b6a9-3275895beaf7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:313bed9d-6e7b-40ce-b6a9-3275895beaf7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:19:14,239 [243] [DEBUG] [app] Ending request: urn:request:fb35931c-2ec7-4260-809b-dfc373e268fb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fb35931c-2ec7-4260-809b-dfc373e268fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:19:14,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:19:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:19:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:14,242 [238] [DEBUG] [app] Starting request: urn:request:7e9ac21a-e2a5-48f8-9768-b6e54ccc11d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:14,242 [235] [DEBUG] [app] Starting request: urn:request:553bd4cc-f85d-4961-8d71-bbea5e57813e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:14,242 [238] [DEBUG] [app] Ending request: urn:request:7e9ac21a-e2a5-48f8-9768-b6e54ccc11d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7e9ac21a-e2a5-48f8-9768-b6e54ccc11d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:19:14,242 [235] [DEBUG] [app] Ending request: urn:request:553bd4cc-f85d-4961-8d71-bbea5e57813e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:553bd4cc-f85d-4961-8d71-bbea5e57813e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:19:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:19:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:19:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:19:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:14,254 [237] [DEBUG] [app] Ending request: urn:request:4030f5a0-8529-4b0e-b419-1033085dacf3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4030f5a0-8529-4b0e-b419-1033085dacf3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:19:14,254 [235] [DEBUG] [app] Ending request: urn:request:b75a9080-1c81-4609-9a6b-41480ca3dde1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b75a9080-1c81-4609-9a6b-41480ca3dde1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:19:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:19:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
namespacegcworker stdout | 2025-02-07 03:19:14,885 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:19:15,315 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:19:15,583 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-web stdout | 2025-02-07 03:19:15,586 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:19:15,589 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:19:15,593 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:19:15,595 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:19:15,598 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 03:19:16,622 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:19:16,626 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:19:16,629 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:19:17,519 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:19:17,522 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:19:17,525 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:19:17,528 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:19:17,532 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:19:17,537 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:19:17,541 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:19:17,543 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:19:17,546 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:19:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:19:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:28.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 03:19:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:28 UTC)" (scheduled at 2025-02-07 03:19:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:19:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:19:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 18, 732921), True, datetime.datetime(2025, 2, 7, 3, 19, 18, 732921), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:19:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:19:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:19:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:19:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:19:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:03.419110+00:00 (in 39.997688 seconds)
repositorygcworker stdout | 2025-02-07 03:19:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:23 UTC)" (scheduled at 2025-02-07 03:19:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:19:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:19:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:19:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:19:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:40.424250+00:00 (in 16.000699 seconds)
gcworker stdout | 2025-02-07 03:19:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:54 UTC)" (scheduled at 2025-02-07 03:19:24.423055+00:00)
gcworker stdout | 2025-02-07 03:19:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:19:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:19:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:19:24,575 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:19:26,541 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:19:27,548 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:19:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:19:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:38.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 03:19:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:38 UTC)" (scheduled at 2025-02-07 03:19:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:19:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:19:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 28, 732906), True, datetime.datetime(2025, 2, 7, 3, 19, 28, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:19:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:19:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:19:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:19:29,235 [236] [DEBUG] [app] Starting request: urn:request:1ae19ca4-2fa0-46ee-b88e-c9f6731d0e05 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:29,235 [237] [DEBUG] [app] Starting request: urn:request:cbeb1349-23b8-48ca-b5d0-4fecd4e45494 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:19:29,238 [243] [DEBUG] [app] Starting request: urn:request:148aa1a7-27c5-4d06-b5ef-7c34b59b0b2a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:19:29,238 [246] [DEBUG] [app] Starting request: urn:request:4a5b774c-f573-4a18-93f5-1a9029a88320 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:19:29,239 [246] [DEBUG] [app] Ending request: urn:request:4a5b774c-f573-4a18-93f5-1a9029a88320 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a5b774c-f573-4a18-93f5-1a9029a88320', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:19:29,239 [243] [DEBUG] [app] Ending request: urn:request:148aa1a7-27c5-4d06-b5ef-7c34b59b0b2a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:148aa1a7-27c5-4d06-b5ef-7c34b59b0b2a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:19:29,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:19:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:29,242 [235] [DEBUG] [app] Starting request: urn:request:c44d32da-651b-4904-91e0-e3fb7634b858 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:29,242 [235] [DEBUG] [app] Ending request: urn:request:c44d32da-651b-4904-91e0-e3fb7634b858 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c44d32da-651b-4904-91e0-e3fb7634b858', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:19:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:29,243 [238] [DEBUG] [app] Starting request: urn:request:48c93ced-aedb-4c5c-9fef-72e8c90d505b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:29,243 [238] [DEBUG] [app] Ending request: urn:request:48c93ced-aedb-4c5c-9fef-72e8c90d505b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48c93ced-aedb-4c5c-9fef-72e8c90d505b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:19:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:19:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:19:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:29,254 [236] [DEBUG] [app] Ending request: urn:request:1ae19ca4-2fa0-46ee-b88e-c9f6731d0e05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ae19ca4-2fa0-46ee-b88e-c9f6731d0e05', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:19:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:19:29,255 [237] [DEBUG] [app] Ending request: urn:request:cbeb1349-23b8-48ca-b5d0-4fecd4e45494 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cbeb1349-23b8-48ca-b5d0-4fecd4e45494', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:19:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:19:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:19:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:32.625945+00:00 (in 1.004362 seconds)
namespacegcworker stdout | 2025-02-07 03:19:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:31 UTC)" (scheduled at 2025-02-07 03:19:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:19:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:19:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 31, 621868), True, datetime.datetime(2025, 2, 7, 3, 19, 31, 621868), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:19:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:19:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:19:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:19:31,843 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:19:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:19:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:31.621131+00:00 (in 58.994747 seconds)
namespacegcworker stdout | 2025-02-07 03:19:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:32 UTC)" (scheduled at 2025-02-07 03:19:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:19:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:19:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:19:32,887 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:19:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:19:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:03.919286+00:00 (in 29.999526 seconds)
buildlogsarchiver stdout | 2025-02-07 03:19:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:03 UTC)" (scheduled at 2025-02-07 03:19:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:19:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 19, 33, 920024), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:19:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:19:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:19:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:03 UTC)" executed successfully
builder stdout | 2025-02-07 03:19:37,507 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:56.713547+00:00 (in 18.997204 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:37 UTC)" (scheduled at 2025-02-07 03:19:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:37 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:19:37,985 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:19:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:19:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:48.732147+00:00 (in 9.999509 seconds)
notificationworker stdout | 2025-02-07 03:19:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:48 UTC)" (scheduled at 2025-02-07 03:19:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:19:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:19:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 38, 732965), True, datetime.datetime(2025, 2, 7, 3, 19, 38, 732965), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:19:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:19:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:19:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:19:39,293 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:19:40,035 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:19:40,182 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:19:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:19:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:54.423055+00:00 (in 13.998302 seconds)
gcworker stdout | 2025-02-07 03:19:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:10 UTC)" (scheduled at 2025-02-07 03:19:40.424250+00:00)
gcworker stdout | 2025-02-07 03:19:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:19:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898080431, None, 1, 0])
gcworker stdout | 2025-02-07 03:19:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:19:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:19:41,192 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:41,434 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:19:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:19:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:11.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 03:19:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:11 UTC)" (scheduled at 2025-02-07 03:19:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:19:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894781836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:19:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:19:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:19:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:19:42,248 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:19:42,354 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:19:42,370 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:19:42,807 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:19:43,211 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:19:43,258 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:19:43,408 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:19:44,235 [235] [DEBUG] [app] Starting request: urn:request:cf49ceb6-f907-4e43-a57f-03e233d7ddfa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:44,235 [238] [DEBUG] [app] Starting request: urn:request:3a78a98c-ff6c-4454-a093-9f9603500b1d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:19:44,238 [244] [DEBUG] [app] Starting request: urn:request:c5a44bb1-3f15-484d-8788-23dfa5cb1dcb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:19:44,238 [239] [DEBUG] [app] Starting request: urn:request:6b7e5598-2f80-4d3b-a3fb-a6d69e80e0bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:19:44,239 [239] [DEBUG] [app] Ending request: urn:request:6b7e5598-2f80-4d3b-a3fb-a6d69e80e0bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b7e5598-2f80-4d3b-a3fb-a6d69e80e0bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:19:44,239 [244] [DEBUG] [app] Ending request: urn:request:c5a44bb1-3f15-484d-8788-23dfa5cb1dcb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5a44bb1-3f15-484d-8788-23dfa5cb1dcb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:19:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:19:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:19:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:44,241 [235] [DEBUG] [app] Starting request: urn:request:885df4c4-511b-4fdd-8cfb-6155861a9fdf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:44,241 [237] [DEBUG] [app] Starting request: urn:request:934d1cec-c7d5-4de1-857e-91b9147a0ae2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:44,241 [235] [DEBUG] [app] Ending request: urn:request:885df4c4-511b-4fdd-8cfb-6155861a9fdf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:885df4c4-511b-4fdd-8cfb-6155861a9fdf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:19:44,242 [237] [DEBUG] [app] Ending request: urn:request:934d1cec-c7d5-4de1-857e-91b9147a0ae2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:934d1cec-c7d5-4de1-857e-91b9147a0ae2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:19:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:19:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:19:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:19:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:44,253 [235] [DEBUG] [app] Ending request: urn:request:cf49ceb6-f907-4e43-a57f-03e233d7ddfa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf49ceb6-f907-4e43-a57f-03e233d7ddfa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:44,253 [238] [DEBUG] [app] Ending request: urn:request:3a78a98c-ff6c-4454-a093-9f9603500b1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a78a98c-ff6c-4454-a093-9f9603500b1d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:19:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:19:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:19:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
namespacegcworker stdout | 2025-02-07 03:19:44,921 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:19:45,331 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:19:45,596 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:19:45,599 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:19:45,601 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:19:45,604 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:19:45,607 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:19:45,619 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:19:46,631 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:19:46,634 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:19:46,637 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:19:47,527 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:19:47,531 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:19:47,535 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:19:47,538 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:19:47,541 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:19:47,546 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:19:47,549 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:19:47,552 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:19:47,554 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:19:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:19:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:50.814677+00:00 (in 2.082029 seconds)
notificationworker stdout | 2025-02-07 03:19:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:58 UTC)" (scheduled at 2025-02-07 03:19:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:19:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:19:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 48, 732930), True, datetime.datetime(2025, 2, 7, 3, 19, 48, 732930), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:19:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:19:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:19:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:19:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:19:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:19:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:58.732147+00:00 (in 7.917007 seconds)
notificationworker stdout | 2025-02-07 03:19:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:50 UTC)" (scheduled at 2025-02-07 03:19:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:19:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:19:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:19:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:19:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:19:59.937899+00:00 (in 6.999766 seconds)
exportactionlogsworker stdout | 2025-02-07 03:19:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:24:52 UTC)" (scheduled at 2025-02-07 03:19:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:19:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 19, 52, 938419), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:19:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 19, 52, 938419), True, datetime.datetime(2025, 2, 7, 3, 19, 52, 938419), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:19:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 19, 52, 938419), True, datetime.datetime(2025, 2, 7, 3, 19, 52, 938419), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 19, 52, 938419), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:19:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:19:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:24:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:19:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:19:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:10.424250+00:00 (in 16.000762 seconds)
gcworker stdout | 2025-02-07 03:19:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:24 UTC)" (scheduled at 2025-02-07 03:19:54.423055+00:00)
gcworker stdout | 2025-02-07 03:19:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:19:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:19:54,611 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:19:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:19:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:56.118855+00:00 (in 59.999462 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:19:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:56 UTC)" (scheduled at 2025-02-07 03:19:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:19:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:19:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:19:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:19:56,563 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:37.715878+00:00 (in 41.001865 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:56 UTC)" (scheduled at 2025-02-07 03:19:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:19:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:56 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:19:57,576 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:19:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:19:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:08.732147+00:00 (in 9.999539 seconds)
notificationworker stdout | 2025-02-07 03:19:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:08 UTC)" (scheduled at 2025-02-07 03:19:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:19:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:19:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 19, 58, 732880), True, datetime.datetime(2025, 2, 7, 3, 19, 58, 732880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:19:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:19:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:19:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:19:59,235 [237] [DEBUG] [app] Starting request: urn:request:d3e230b3-2842-4541-a362-9c4b12942cde (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:59,235 [235] [DEBUG] [app] Starting request: urn:request:3f43bf84-4342-477d-92af-9fbc4886d749 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:19:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:19:59,238 [246] [DEBUG] [app] Starting request: urn:request:afe558c4-2d1e-4831-8362-505e1a0ad45b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:19:59,239 [246] [DEBUG] [app] Ending request: urn:request:afe558c4-2d1e-4831-8362-505e1a0ad45b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:afe558c4-2d1e-4831-8362-505e1a0ad45b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:19:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:19:59,240 [245] [DEBUG] [app] Starting request: urn:request:f4feab4f-5e0c-4c07-9b71-d0429c47984a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:19:59,241 [245] [DEBUG] [app] Ending request: urn:request:f4feab4f-5e0c-4c07-9b71-d0429c47984a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4feab4f-5e0c-4c07-9b71-d0429c47984a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 162 0.003)
gunicorn-registry stdout | 2025-02-07 03:19:59,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:59,242 [235] [DEBUG] [app] Starting request: urn:request:789c1776-feaf-4841-9e49-b200e199bc9c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:59,242 [235] [DEBUG] [app] Ending request: urn:request:789c1776-feaf-4841-9e49-b200e199bc9c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:789c1776-feaf-4841-9e49-b200e199bc9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:19:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:19:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:59,244 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:19:59,245 [235] [DEBUG] [app] Starting request: urn:request:15534e89-8f84-4540-8a6c-6bb7d8370380 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:19:59,246 [235] [DEBUG] [app] Ending request: urn:request:15534e89-8f84-4540-8a6c-6bb7d8370380 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:15534e89-8f84-4540-8a6c-6bb7d8370380', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:19:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:19:59,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:19:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:19:59,246 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:19:59,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:19:59,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:19:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:59,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:19:59,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:19:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:19:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:59,254 [237] [DEBUG] [app] Ending request: urn:request:d3e230b3-2842-4541-a362-9c4b12942cde (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d3e230b3-2842-4541-a362-9c4b12942cde', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:19:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:19:59,255 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 03:19:59,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:19:59,257 [235] [DEBUG] [app] Ending request: urn:request:3f43bf84-4342-477d-92af-9fbc4886d749 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f43bf84-4342-477d-92af-9fbc4886d749', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:19:59,258 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:19:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.023)
gunicorn-web stdout | 2025-02-07 03:19:59,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:19:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:19:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:19:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:04.936688+00:00 (in 4.998303 seconds)
exportactionlogsworker stdout | 2025-02-07 03:19:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:59 UTC)" (scheduled at 2025-02-07 03:19:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:19:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:19:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:20:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:20:01,879 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:20:02,923 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:20:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:20:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:23.420886+00:00 (in 20.001290 seconds)
repositorygcworker stdout | 2025-02-07 03:20:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:03 UTC)" (scheduled at 2025-02-07 03:20:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:20:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:20:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 3, 419916), True, datetime.datetime(2025, 2, 7, 3, 20, 3, 419916), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:20:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:20:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:20:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:20:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:20:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:33.919286+00:00 (in 29.999533 seconds)
buildlogsarchiver stdout | 2025-02-07 03:20:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:33 UTC)" (scheduled at 2025-02-07 03:20:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:20:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 20, 3, 920031), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:20:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:20:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:20:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:20:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:20:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:59.937899+00:00 (in 55.000746 seconds)
exportactionlogsworker stdout | 2025-02-07 03:20:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:04 UTC)" (scheduled at 2025-02-07 03:20:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:20:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:20:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 4, 937487), True, datetime.datetime(2025, 2, 7, 3, 20, 4, 937487), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:20:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:20:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:20:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:20:07,543 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:20:08,012 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:20:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:20:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:18.732147+00:00 (in 9.999389 seconds)
notificationworker stdout | 2025-02-07 03:20:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:18 UTC)" (scheduled at 2025-02-07 03:20:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:20:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:20:08,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 8, 733267), True, datetime.datetime(2025, 2, 7, 3, 20, 8, 733267), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:20:08,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:20:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:20:08,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:20:09,322 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:20:10,071 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:20:10,194 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:20:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:20:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:24.423055+00:00 (in 13.998366 seconds)
gcworker stdout | 2025-02-07 03:20:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:40 UTC)" (scheduled at 2025-02-07 03:20:10.424250+00:00)
gcworker stdout | 2025-02-07 03:20:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:20:10,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898110431, None, 1, 0])
gcworker stdout | 2025-02-07 03:20:10,434 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:20:10,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:20:11,218 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:11,471 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:20:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:20:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:41.831521+00:00 (in 29.999528 seconds)
autopruneworker stdout | 2025-02-07 03:20:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:41 UTC)" (scheduled at 2025-02-07 03:20:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:20:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894811836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:20:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:20:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:20:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:20:12,274 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:20:12,387 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:20:12,390 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:20:12,831 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:20:13,247 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:20:13,294 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:20:13,444 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:20:14,235 [235] [DEBUG] [app] Starting request: urn:request:8898846b-5aac-4acc-8090-e022ab8b6364 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:14,235 [238] [DEBUG] [app] Starting request: urn:request:d2291529-2a1c-4e79-9c2d-6cb4642bc2f1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:20:14,239 [246] [DEBUG] [app] Starting request: urn:request:c728d58e-ca28-404e-bbf6-456242f40b32 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:14,239 [239] [DEBUG] [app] Starting request: urn:request:494d5ef5-c67f-4c9e-af00-e0c774014f73 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:14,239 [246] [DEBUG] [app] Ending request: urn:request:c728d58e-ca28-404e-bbf6-456242f40b32 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c728d58e-ca28-404e-bbf6-456242f40b32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:20:14,239 [239] [DEBUG] [app] Ending request: urn:request:494d5ef5-c67f-4c9e-af00-e0c774014f73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:494d5ef5-c67f-4c9e-af00-e0c774014f73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:20:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:20:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:20:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:14,243 [236] [DEBUG] [app] Starting request: urn:request:b33b4cf4-290e-4b94-ae32-fcd2e33eb5bb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:20:14,243 [237] [DEBUG] [app] Starting request: urn:request:39e3e192-80b8-418e-8a65-be995c4d83e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:20:14,243 [237] [DEBUG] [app] Ending request: urn:request:39e3e192-80b8-418e-8a65-be995c4d83e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39e3e192-80b8-418e-8a65-be995c4d83e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:20:14,243 [236] [DEBUG] [app] Ending request: urn:request:b33b4cf4-290e-4b94-ae32-fcd2e33eb5bb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b33b4cf4-290e-4b94-ae32-fcd2e33eb5bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:20:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:14,255 [235] [DEBUG] [app] Ending request: urn:request:8898846b-5aac-4acc-8090-e022ab8b6364 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8898846b-5aac-4acc-8090-e022ab8b6364', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:14,255 [238] [DEBUG] [app] Ending request: urn:request:d2291529-2a1c-4e79-9c2d-6cb4642bc2f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d2291529-2a1c-4e79-9c2d-6cb4642bc2f1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:20:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:20:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:20:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
namespacegcworker stdout | 2025-02-07 03:20:14,936 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:20:15,354 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:20:15,604 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:20:15,607 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:20:15,610 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:20:15,614 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:20:15,617 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:20:15,655 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:20:16,639 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:20:16,641 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:20:16,644 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:20:17,536 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:20:17,539 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:20:17,542 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:20:17,544 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:20:17,548 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:20:17,553 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:20:17,557 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:20:17,560 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:20:17,562 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:20:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:20:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:28.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 03:20:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:28 UTC)" (scheduled at 2025-02-07 03:20:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:20:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:20:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 18, 732900), True, datetime.datetime(2025, 2, 7, 3, 20, 18, 732900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:20:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:20:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:20:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:20:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:20:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:03.419110+00:00 (in 39.997716 seconds)
repositorygcworker stdout | 2025-02-07 03:20:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:23 UTC)" (scheduled at 2025-02-07 03:20:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:20:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:20:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:20:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:20:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:40.424250+00:00 (in 16.000748 seconds)
gcworker stdout | 2025-02-07 03:20:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:54 UTC)" (scheduled at 2025-02-07 03:20:24.423055+00:00)
gcworker stdout | 2025-02-07 03:20:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:20:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:20:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:20:24,634 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:20:26,578 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:20:27,613 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:20:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:20:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:38.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 03:20:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:38 UTC)" (scheduled at 2025-02-07 03:20:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:20:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:20:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 28, 732869), True, datetime.datetime(2025, 2, 7, 3, 20, 28, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:20:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:20:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:20:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:20:29,235 [238] [DEBUG] [app] Starting request: urn:request:9d3eb516-9a66-49cd-8161-2ac4370bc49a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:29,235 [237] [DEBUG] [app] Starting request: urn:request:89f273b2-3db9-42bf-b1c9-6fa0d7cd87b7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:20:29,238 [244] [DEBUG] [app] Starting request: urn:request:937251e2-9f8b-4f5b-93a7-259e0a998115 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:29,238 [239] [DEBUG] [app] Starting request: urn:request:5cb9fe3d-e09e-4dd6-863e-430c6f59de03 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:29,239 [244] [DEBUG] [app] Ending request: urn:request:937251e2-9f8b-4f5b-93a7-259e0a998115 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:937251e2-9f8b-4f5b-93a7-259e0a998115', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:20:29,239 [239] [DEBUG] [app] Ending request: urn:request:5cb9fe3d-e09e-4dd6-863e-430c6f59de03 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cb9fe3d-e09e-4dd6-863e-430c6f59de03', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:20:29,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:20:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:29,241 [237] [DEBUG] [app] Starting request: urn:request:778d13cd-7a1d-4e88-b299-f2ff8f35b6d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:20:29,242 [236] [DEBUG] [app] Starting request: urn:request:4a1b679b-ed84-405d-a276-b27e3b4b3800 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:20:29,242 [237] [DEBUG] [app] Ending request: urn:request:778d13cd-7a1d-4e88-b299-f2ff8f35b6d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:778d13cd-7a1d-4e88-b299-f2ff8f35b6d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:20:29,242 [236] [DEBUG] [app] Ending request: urn:request:4a1b679b-ed84-405d-a276-b27e3b4b3800 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a1b679b-ed84-405d-a276-b27e3b4b3800', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:20:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:20:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:20:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:29,254 [237] [DEBUG] [app] Ending request: urn:request:89f273b2-3db9-42bf-b1c9-6fa0d7cd87b7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:89f273b2-3db9-42bf-b1c9-6fa0d7cd87b7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:29,254 [238] [DEBUG] [app] Ending request: urn:request:9d3eb516-9a66-49cd-8161-2ac4370bc49a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d3eb516-9a66-49cd-8161-2ac4370bc49a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:20:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:20:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:20:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:20:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:20:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:32.625945+00:00 (in 1.004338 seconds)
namespacegcworker stdout | 2025-02-07 03:20:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:31 UTC)" (scheduled at 2025-02-07 03:20:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:20:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:20:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 31, 621882), True, datetime.datetime(2025, 2, 7, 3, 20, 31, 621882), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:20:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:20:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:20:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:20:31,901 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:20:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:20:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:31.621131+00:00 (in 58.994725 seconds)
namespacegcworker stdout | 2025-02-07 03:20:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:32 UTC)" (scheduled at 2025-02-07 03:20:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:20:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:20:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:20:32,937 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:20:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:20:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:03.919286+00:00 (in 29.999514 seconds)
buildlogsarchiver stdout | 2025-02-07 03:20:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:03 UTC)" (scheduled at 2025-02-07 03:20:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:20:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 20, 33, 920050), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:20:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:20:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:20:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:03 UTC)" executed successfully
builder stdout | 2025-02-07 03:20:37,580 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:56.713547+00:00 (in 18.997188 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:37 UTC)" (scheduled at 2025-02-07 03:20:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:37 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:20:38,042 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:20:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:20:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:48.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 03:20:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:48 UTC)" (scheduled at 2025-02-07 03:20:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:20:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:20:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 38, 732878), True, datetime.datetime(2025, 2, 7, 3, 20, 38, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:20:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:20:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:20:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:20:39,331 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:20:40,107 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:20:40,210 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:20:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:20:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:54.423055+00:00 (in 13.998385 seconds)
gcworker stdout | 2025-02-07 03:20:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:10 UTC)" (scheduled at 2025-02-07 03:20:40.424250+00:00)
gcworker stdout | 2025-02-07 03:20:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:20:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898140431, None, 1, 0])
gcworker stdout | 2025-02-07 03:20:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:20:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:20:41,254 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:41,507 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:20:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:20:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:11.831521+00:00 (in 29.999519 seconds)
autopruneworker stdout | 2025-02-07 03:20:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:11 UTC)" (scheduled at 2025-02-07 03:20:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:20:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894841836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:20:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:20:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:20:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:20:42,310 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:20:42,406 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:20:42,422 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:20:42,845 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:20:43,258 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:20:43,331 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:20:43,461 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:20:44,236 [238] [DEBUG] [app] Starting request: urn:request:39e274db-ade6-4738-b820-d0624755611c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:44,236 [235] [DEBUG] [app] Starting request: urn:request:cbba20d0-f370-4cce-8f15-95d26cb264fb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:44,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:20:44,240 [239] [DEBUG] [app] Starting request: urn:request:48081020-e809-435d-a041-30fc4a1286aa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:44,240 [245] [DEBUG] [app] Starting request: urn:request:177c168a-04d8-4e20-b79e-ad185801c118 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:44,240 [239] [DEBUG] [app] Ending request: urn:request:48081020-e809-435d-a041-30fc4a1286aa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48081020-e809-435d-a041-30fc4a1286aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:20:44,240 [245] [DEBUG] [app] Ending request: urn:request:177c168a-04d8-4e20-b79e-ad185801c118 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:177c168a-04d8-4e20-b79e-ad185801c118', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:20:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:20:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:44,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:44,243 [238] [DEBUG] [app] Starting request: urn:request:d589c678-2036-4d45-920a-dbd25f90f7c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:20:44,243 [238] [DEBUG] [app] Ending request: urn:request:d589c678-2036-4d45-920a-dbd25f90f7c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d589c678-2036-4d45-920a-dbd25f90f7c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:20:44,243 [237] [DEBUG] [app] Starting request: urn:request:f5b03fef-76a9-4f3a-9aea-3be13c6b8877 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:20:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:44,243 [237] [DEBUG] [app] Ending request: urn:request:f5b03fef-76a9-4f3a-9aea-3be13c6b8877 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f5b03fef-76a9-4f3a-9aea-3be13c6b8877', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:20:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:20:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:44,255 [238] [DEBUG] [app] Ending request: urn:request:39e274db-ade6-4738-b820-d0624755611c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39e274db-ade6-4738-b820-d0624755611c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:20:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:20:44,255 [235] [DEBUG] [app] Ending request: urn:request:cbba20d0-f370-4cce-8f15-95d26cb264fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cbba20d0-f370-4cce-8f15-95d26cb264fb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:20:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:20:44,944 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:20:45,390 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:20:45,611 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:20:45,614 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:20:45,617 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:20:45,622 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:20:45,626 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:20:45,692 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:20:46,646 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:20:46,649 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:20:46,652 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:20:47,548 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:20:47,551 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:20:47,553 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:20:47,556 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:20:47,558 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:20:47,561 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:20:47,564 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:20:47,569 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:20:47,572 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:20:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:20:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:50.814677+00:00 (in 2.082093 seconds)
notificationworker stdout | 2025-02-07 03:20:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:58 UTC)" (scheduled at 2025-02-07 03:20:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:20:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:20:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 48, 732860), True, datetime.datetime(2025, 2, 7, 3, 20, 48, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:20:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:20:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:20:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:20:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:20:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:20:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:20:58.732147+00:00 (in 7.916969 seconds)
notificationworker stdout | 2025-02-07 03:20:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:50 UTC)" (scheduled at 2025-02-07 03:20:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:20:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:20:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:20:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:20:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:10.424250+00:00 (in 16.000709 seconds)
gcworker stdout | 2025-02-07 03:20:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:24 UTC)" (scheduled at 2025-02-07 03:20:54.423055+00:00)
gcworker stdout | 2025-02-07 03:20:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:20:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:20:54,671 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:20:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:20:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:56.118855+00:00 (in 59.999499 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:20:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:56 UTC)" (scheduled at 2025-02-07 03:20:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:20:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:20:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:20:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:20:56,602 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:37.715878+00:00 (in 41.001884 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:56 UTC)" (scheduled at 2025-02-07 03:20:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:20:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:56 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:20:57,633 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:20:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:20:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:08.732147+00:00 (in 9.999501 seconds)
notificationworker stdout | 2025-02-07 03:20:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:08 UTC)" (scheduled at 2025-02-07 03:20:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:20:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:20:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 20, 58, 732939), True, datetime.datetime(2025, 2, 7, 3, 20, 58, 732939), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:20:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:20:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:20:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:20:59,235 [238] [DEBUG] [app] Starting request: urn:request:b675b06d-a9a9-4e00-874d-8ca1c2239ef8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:59,235 [237] [DEBUG] [app] Starting request: urn:request:dfec4bec-9d47-4e96-afc6-07c2786c3bc4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:20:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:20:59,238 [244] [DEBUG] [app] Starting request: urn:request:429632ff-4e36-4eea-9b8a-1da70f9acafe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:59,238 [239] [DEBUG] [app] Starting request: urn:request:594c3cd4-6a19-451e-a54f-db706c06b61f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:20:59,239 [244] [DEBUG] [app] Ending request: urn:request:429632ff-4e36-4eea-9b8a-1da70f9acafe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:429632ff-4e36-4eea-9b8a-1da70f9acafe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:20:59,239 [239] [DEBUG] [app] Ending request: urn:request:594c3cd4-6a19-451e-a54f-db706c06b61f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:594c3cd4-6a19-451e-a54f-db706c06b61f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:20:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:20:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:20:59,241 [238] [DEBUG] [app] Starting request: urn:request:b8949e8c-57cb-40d7-bb53-8e0b95e316c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:20:59,241 [237] [DEBUG] [app] Starting request: urn:request:0818af98-26f3-4a90-a4be-5bf59f569b7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:20:59,241 [237] [DEBUG] [app] Ending request: urn:request:0818af98-26f3-4a90-a4be-5bf59f569b7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0818af98-26f3-4a90-a4be-5bf59f569b7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:20:59,241 [238] [DEBUG] [app] Ending request: urn:request:b8949e8c-57cb-40d7-bb53-8e0b95e316c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8949e8c-57cb-40d7-bb53-8e0b95e316c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:20:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:20:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:20:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:20:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:20:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:20:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:20:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:20:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:20:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:20:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:20:59,253 [238] [DEBUG] [app] Ending request: urn:request:b675b06d-a9a9-4e00-874d-8ca1c2239ef8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b675b06d-a9a9-4e00-874d-8ca1c2239ef8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:20:59,253 [237] [DEBUG] [app] Ending request: urn:request:dfec4bec-9d47-4e96-afc6-07c2786c3bc4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dfec4bec-9d47-4e96-afc6-07c2786c3bc4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:20:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:20:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:20:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:20:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:20:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:20:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:20:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:04.936688+00:00 (in 4.998286 seconds)
exportactionlogsworker stdout | 2025-02-07 03:20:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:59 UTC)" (scheduled at 2025-02-07 03:20:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:20:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:20:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:21:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:21:01,922 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:21:02,950 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:21:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:21:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:23.420886+00:00 (in 20.001271 seconds)
repositorygcworker stdout | 2025-02-07 03:21:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:03 UTC)" (scheduled at 2025-02-07 03:21:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:21:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:21:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 3, 419934), True, datetime.datetime(2025, 2, 7, 3, 21, 3, 419934), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:21:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:21:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:21:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:21:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:21:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:33.919286+00:00 (in 29.999519 seconds)
buildlogsarchiver stdout | 2025-02-07 03:21:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:33 UTC)" (scheduled at 2025-02-07 03:21:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:21:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 21, 3, 920043), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:21:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:21:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:21:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:21:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:21:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:59.937899+00:00 (in 55.000769 seconds)
exportactionlogsworker stdout | 2025-02-07 03:21:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:04 UTC)" (scheduled at 2025-02-07 03:21:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:21:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:21:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 4, 937449), True, datetime.datetime(2025, 2, 7, 3, 21, 4, 937449), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:21:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:21:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:21:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:21:07,590 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:21:08,061 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:21:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:21:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:18.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 03:21:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:18 UTC)" (scheduled at 2025-02-07 03:21:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:21:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:21:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 8, 732904), True, datetime.datetime(2025, 2, 7, 3, 21, 8, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:21:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:21:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:21:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:21:09,358 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:21:10,141 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:21:10,234 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:21:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:21:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:24.423055+00:00 (in 13.998275 seconds)
gcworker stdout | 2025-02-07 03:21:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:40 UTC)" (scheduled at 2025-02-07 03:21:10.424250+00:00)
gcworker stdout | 2025-02-07 03:21:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:21:10,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898170431, None, 1, 0])
gcworker stdout | 2025-02-07 03:21:10,434 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:21:10,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:21:11,291 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:11,544 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:21:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:21:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:41.831521+00:00 (in 29.999528 seconds)
autopruneworker stdout | 2025-02-07 03:21:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:41 UTC)" (scheduled at 2025-02-07 03:21:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:21:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894871836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:21:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:21:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:21:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:21:12,332 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:21:12,443 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:21:12,455 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:21:12,853 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:21:13,295 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:21:13,367 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:21:13,480 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:21:14,236 [236] [DEBUG] [app] Starting request: urn:request:309e27a6-6330-4b87-9377-837fc3c37d53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:14,236 [237] [DEBUG] [app] Starting request: urn:request:b874cb58-f840-49ec-b313-865c6b4924bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:21:14,239 [239] [DEBUG] [app] Starting request: urn:request:0d685953-6713-47dd-81a4-bd382211ac90 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:14,239 [246] [DEBUG] [app] Starting request: urn:request:b41069f9-c3b7-4627-9d34-0195f74caad9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:14,239 [239] [DEBUG] [app] Ending request: urn:request:0d685953-6713-47dd-81a4-bd382211ac90 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d685953-6713-47dd-81a4-bd382211ac90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:21:14,239 [246] [DEBUG] [app] Ending request: urn:request:b41069f9-c3b7-4627-9d34-0195f74caad9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b41069f9-c3b7-4627-9d34-0195f74caad9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:21:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:21:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:14,242 [237] [DEBUG] [app] Starting request: urn:request:adb64542-994e-4103-a450-6f33a8d03abd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:21:14,242 [237] [DEBUG] [app] Ending request: urn:request:adb64542-994e-4103-a450-6f33a8d03abd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:adb64542-994e-4103-a450-6f33a8d03abd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:21:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:14,242 [238] [DEBUG] [app] Starting request: urn:request:0b079916-28a9-4f8e-a033-399f241ad56d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:21:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:14,243 [238] [DEBUG] [app] Ending request: urn:request:0b079916-28a9-4f8e-a033-399f241ad56d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b079916-28a9-4f8e-a033-399f241ad56d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:21:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:21:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:21:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:21:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:14,255 [237] [DEBUG] [app] Ending request: urn:request:b874cb58-f840-49ec-b313-865c6b4924bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b874cb58-f840-49ec-b313-865c6b4924bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:21:14,255 [236] [DEBUG] [app] Ending request: urn:request:309e27a6-6330-4b87-9377-837fc3c37d53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:309e27a6-6330-4b87-9377-837fc3c37d53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:21:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:21:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:21:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:21:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:21:14,958 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:21:15,426 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:21:15,620 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:21:15,622 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:21:15,625 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:21:15,628 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:21:15,633 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:21:15,724 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:21:16,653 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:21:16,657 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:21:16,659 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:21:17,559 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:21:17,562 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:21:17,565 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:21:17,568 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:21:17,572 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:21:17,575 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:21:17,577 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:21:17,580 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:21:17,583 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:21:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:21:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:28.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 03:21:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:28 UTC)" (scheduled at 2025-02-07 03:21:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:21:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:21:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 18, 732860), True, datetime.datetime(2025, 2, 7, 3, 21, 18, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:21:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:21:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:21:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:21:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:21:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:03.419110+00:00 (in 39.997697 seconds)
repositorygcworker stdout | 2025-02-07 03:21:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:23 UTC)" (scheduled at 2025-02-07 03:21:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:21:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:21:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:21:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:21:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:40.424250+00:00 (in 16.000722 seconds)
gcworker stdout | 2025-02-07 03:21:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:54 UTC)" (scheduled at 2025-02-07 03:21:24.423055+00:00)
gcworker stdout | 2025-02-07 03:21:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:21:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:21:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:21:24,687 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:21:26,638 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:21:27,670 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:21:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:21:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:38.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 03:21:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:38 UTC)" (scheduled at 2025-02-07 03:21:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:21:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:21:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 28, 732882), True, datetime.datetime(2025, 2, 7, 3, 21, 28, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:21:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:21:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:21:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:21:29,235 [238] [DEBUG] [app] Starting request: urn:request:1c1da92f-5bba-4623-980d-053e9c6c272b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:29,235 [237] [DEBUG] [app] Starting request: urn:request:55019190-f04a-4dd2-bb3a-4c6c3cc3e2a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:21:29,238 [239] [DEBUG] [app] Starting request: urn:request:13432935-055e-4c9d-b9b5-5f1f5ec7e07d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:29,239 [245] [DEBUG] [app] Starting request: urn:request:307d53a6-2e34-4916-b864-9f0f49af64a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:29,239 [239] [DEBUG] [app] Ending request: urn:request:13432935-055e-4c9d-b9b5-5f1f5ec7e07d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13432935-055e-4c9d-b9b5-5f1f5ec7e07d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:21:29,239 [245] [DEBUG] [app] Ending request: urn:request:307d53a6-2e34-4916-b864-9f0f49af64a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:307d53a6-2e34-4916-b864-9f0f49af64a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:21:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:21:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:29,241 [238] [DEBUG] [app] Starting request: urn:request:de06d2d7-a04b-4b80-aa03-163e7ece3d37 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:21:29,242 [238] [DEBUG] [app] Ending request: urn:request:de06d2d7-a04b-4b80-aa03-163e7ece3d37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de06d2d7-a04b-4b80-aa03-163e7ece3d37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:21:29,242 [236] [DEBUG] [app] Starting request: urn:request:7b221c97-f22b-4958-8167-6de8c62bc07f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:21:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:21:29,242 [236] [DEBUG] [app] Ending request: urn:request:7b221c97-f22b-4958-8167-6de8c62bc07f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7b221c97-f22b-4958-8167-6de8c62bc07f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:21:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:21:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:21:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:21:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:29,253 [238] [DEBUG] [app] Ending request: urn:request:1c1da92f-5bba-4623-980d-053e9c6c272b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c1da92f-5bba-4623-980d-053e9c6c272b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:21:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:21:29,254 [237] [DEBUG] [app] Ending request: urn:request:55019190-f04a-4dd2-bb3a-4c6c3cc3e2a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55019190-f04a-4dd2-bb3a-4c6c3cc3e2a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:21:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:21:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:21:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:21:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:21:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:32.625945+00:00 (in 1.004336 seconds)
namespacegcworker stdout | 2025-02-07 03:21:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:31 UTC)" (scheduled at 2025-02-07 03:21:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:21:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:21:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 31, 621923), True, datetime.datetime(2025, 2, 7, 3, 21, 31, 621923), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:21:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:21:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:21:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:21:31,946 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:21:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:21:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:13.624301+00:00 (in 40.997881 seconds)
namespacegcworker stdout | 2025-02-07 03:21:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:32 UTC)" (scheduled at 2025-02-07 03:21:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:21:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:21:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:21:32,986 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:21:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:21:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:03.919286+00:00 (in 29.999561 seconds)
buildlogsarchiver stdout | 2025-02-07 03:21:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:03 UTC)" (scheduled at 2025-02-07 03:21:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:21:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 21, 33, 919976), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:21:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:21:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:21:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:03 UTC)" executed successfully
builder stdout | 2025-02-07 03:21:37,623 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:56.713547+00:00 (in 18.997191 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:37 UTC)" (scheduled at 2025-02-07 03:21:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:37 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:21:38,078 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:21:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:21:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:48.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 03:21:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:48 UTC)" (scheduled at 2025-02-07 03:21:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:21:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:21:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 38, 732895), True, datetime.datetime(2025, 2, 7, 3, 21, 38, 732895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:21:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:21:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:21:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:21:39,379 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:21:40,177 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:21:40,247 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:21:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:21:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:54.423055+00:00 (in 13.998375 seconds)
gcworker stdout | 2025-02-07 03:21:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:10 UTC)" (scheduled at 2025-02-07 03:21:40.424250+00:00)
gcworker stdout | 2025-02-07 03:21:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:21:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898200431, None, 1, 0])
gcworker stdout | 2025-02-07 03:21:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:21:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:21:41,327 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:41,579 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:21:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:21:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:11.831521+00:00 (in 29.999537 seconds)
autopruneworker stdout | 2025-02-07 03:21:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:11 UTC)" (scheduled at 2025-02-07 03:21:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:21:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894901836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:21:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:21:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:21:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:21:42,368 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:21:42,479 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:21:42,490 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:21:42,869 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:21:43,331 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:21:43,404 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:21:43,516 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:21:44,236 [236] [DEBUG] [app] Starting request: urn:request:a39de20b-042f-44a5-b02a-b10be381e74d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:44,236 [238] [DEBUG] [app] Starting request: urn:request:339ad086-6c24-4066-9e9a-8cbd25182ad5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:21:44,239 [239] [DEBUG] [app] Starting request: urn:request:280bacda-75e6-4d85-9502-cc3a0cbed95b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:44,239 [246] [DEBUG] [app] Starting request: urn:request:9a2e7761-08d6-4871-9a08-a8ff6d6f3129 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:44,240 [239] [DEBUG] [app] Ending request: urn:request:280bacda-75e6-4d85-9502-cc3a0cbed95b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:280bacda-75e6-4d85-9502-cc3a0cbed95b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:21:44,240 [246] [DEBUG] [app] Ending request: urn:request:9a2e7761-08d6-4871-9a08-a8ff6d6f3129 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a2e7761-08d6-4871-9a08-a8ff6d6f3129', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:21:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:21:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:21:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:44,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:44,244 [238] [DEBUG] [app] Starting request: urn:request:49280f44-3a2e-4fe3-ae1f-827109af7b89 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:21:44,244 [238] [DEBUG] [app] Ending request: urn:request:49280f44-3a2e-4fe3-ae1f-827109af7b89 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:49280f44-3a2e-4fe3-ae1f-827109af7b89', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:21:44,244 [235] [DEBUG] [app] Starting request: urn:request:b1b3da71-1110-43f3-a78a-a37f0a31bead (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:21:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:21:44,245 [235] [DEBUG] [app] Ending request: urn:request:b1b3da71-1110-43f3-a78a-a37f0a31bead (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b1b3da71-1110-43f3-a78a-a37f0a31bead', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:21:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:21:44,245 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:44,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:21:44,246 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:44,246 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:21:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:44,250 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:44,250 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:44,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:44,255 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:44,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:44,256 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:44,257 [238] [DEBUG] [app] Ending request: urn:request:339ad086-6c24-4066-9e9a-8cbd25182ad5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:339ad086-6c24-4066-9e9a-8cbd25182ad5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:21:44,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:21:44,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.023)
gunicorn-web stdout | 2025-02-07 03:21:44,258 [236] [DEBUG] [app] Ending request: urn:request:a39de20b-042f-44a5-b02a-b10be381e74d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a39de20b-042f-44a5-b02a-b10be381e74d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:21:44,258 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:21:44,259 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
namespacegcworker stdout | 2025-02-07 03:21:44,995 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:21:45,458 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:21:45,628 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:21:45,631 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:21:45,634 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:21:45,637 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:21:45,639 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:21:45,760 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:21:46,661 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:21:46,665 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:21:46,667 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:21:47,568 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:21:47,572 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:21:47,575 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:21:47,578 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:21:47,584 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:21:47,587 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:21:47,590 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:21:47,593 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:21:47,595 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:21:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:21:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:50.814677+00:00 (in 2.082083 seconds)
notificationworker stdout | 2025-02-07 03:21:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:58 UTC)" (scheduled at 2025-02-07 03:21:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:21:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:21:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 48, 732887), True, datetime.datetime(2025, 2, 7, 3, 21, 48, 732887), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:21:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:21:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:21:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:21:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:21:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:21:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:21:58.732147+00:00 (in 7.917007 seconds)
notificationworker stdout | 2025-02-07 03:21:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:50 UTC)" (scheduled at 2025-02-07 03:21:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:21:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:21:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:21:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:21:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:10.424250+00:00 (in 16.000694 seconds)
gcworker stdout | 2025-02-07 03:21:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:24 UTC)" (scheduled at 2025-02-07 03:21:54.423055+00:00)
gcworker stdout | 2025-02-07 03:21:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:21:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:21:54,721 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:21:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:21:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:56.118855+00:00 (in 59.999531 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:21:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:56 UTC)" (scheduled at 2025-02-07 03:21:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:21:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:21:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:21:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:21:56,668 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:37.715878+00:00 (in 41.001890 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:56 UTC)" (scheduled at 2025-02-07 03:21:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:21:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:56 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:21:57,692 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:21:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:21:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:08.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 03:21:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:08 UTC)" (scheduled at 2025-02-07 03:21:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:21:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:21:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 21, 58, 732851), True, datetime.datetime(2025, 2, 7, 3, 21, 58, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:21:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:21:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:21:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:21:59,235 [236] [DEBUG] [app] Starting request: urn:request:ee810f36-4404-4684-840a-553c2ee88179 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:59,235 [237] [DEBUG] [app] Starting request: urn:request:7471cdb8-80c7-4626-b705-a3d5042c28f1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:21:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:21:59,238 [244] [DEBUG] [app] Starting request: urn:request:bec7a2cb-8c39-45a0-87ea-280f9b7de7e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:59,238 [246] [DEBUG] [app] Starting request: urn:request:f5760092-23c9-45f4-b981-7aa18eca40f5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:21:59,239 [244] [DEBUG] [app] Ending request: urn:request:bec7a2cb-8c39-45a0-87ea-280f9b7de7e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bec7a2cb-8c39-45a0-87ea-280f9b7de7e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:21:59,239 [246] [DEBUG] [app] Ending request: urn:request:f5760092-23c9-45f4-b981-7aa18eca40f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5760092-23c9-45f4-b981-7aa18eca40f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:21:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:21:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:21:59,242 [237] [DEBUG] [app] Starting request: urn:request:e5dab577-66c6-443b-b7b6-10886966b58d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:21:59,242 [237] [DEBUG] [app] Ending request: urn:request:e5dab577-66c6-443b-b7b6-10886966b58d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5dab577-66c6-443b-b7b6-10886966b58d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:21:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:59,242 [235] [DEBUG] [app] Starting request: urn:request:58682748-4274-4c75-b895-6d7fa3703721 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:21:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:59,243 [235] [DEBUG] [app] Ending request: urn:request:58682748-4274-4c75-b895-6d7fa3703721 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58682748-4274-4c75-b895-6d7fa3703721', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:21:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:21:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:21:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:21:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:21:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:21:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:21:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:21:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:21:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:21:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:21:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:21:59,254 [237] [DEBUG] [app] Ending request: urn:request:7471cdb8-80c7-4626-b705-a3d5042c28f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7471cdb8-80c7-4626-b705-a3d5042c28f1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:21:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:21:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:21:59,254 [236] [DEBUG] [app] Ending request: urn:request:ee810f36-4404-4684-840a-553c2ee88179 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee810f36-4404-4684-840a-553c2ee88179', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:21:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:21:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:21:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:21:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 03:21:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:21:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:04.936688+00:00 (in 4.998331 seconds)
exportactionlogsworker stdout | 2025-02-07 03:21:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:59 UTC)" (scheduled at 2025-02-07 03:21:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:21:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:21:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:22:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:22:01,978 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:22:03,001 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:22:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:22:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:23.420886+00:00 (in 20.001320 seconds)
repositorygcworker stdout | 2025-02-07 03:22:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:03 UTC)" (scheduled at 2025-02-07 03:22:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:22:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:22:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 3, 419840), True, datetime.datetime(2025, 2, 7, 3, 22, 3, 419840), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:22:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:22:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:22:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:22:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:22:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:33.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 03:22:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:33 UTC)" (scheduled at 2025-02-07 03:22:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:22:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 22, 3, 919976), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:22:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:22:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:22:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:22:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:22:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:59.937899+00:00 (in 55.000763 seconds)
exportactionlogsworker stdout | 2025-02-07 03:22:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:04 UTC)" (scheduled at 2025-02-07 03:22:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:22:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:22:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 4, 937499), True, datetime.datetime(2025, 2, 7, 3, 22, 4, 937499), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:22:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:22:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:22:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:22:07,655 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:22:08,115 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:22:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:18.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 03:22:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:18 UTC)" (scheduled at 2025-02-07 03:22:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:22:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:22:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 8, 732898), True, datetime.datetime(2025, 2, 7, 3, 22, 8, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:22:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:22:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:22:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:22:09,387 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:22:10,212 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:22:10,283 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:22:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:22:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:24.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 03:22:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:40 UTC)" (scheduled at 2025-02-07 03:22:10.424250+00:00)
gcworker stdout | 2025-02-07 03:22:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:22:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898230430, None, 1, 0])
gcworker stdout | 2025-02-07 03:22:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:22:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:22:11,364 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:11,615 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:22:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:22:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:41.831521+00:00 (in 29.999513 seconds)
autopruneworker stdout | 2025-02-07 03:22:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:41 UTC)" (scheduled at 2025-02-07 03:22:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:22:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894931836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:22:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:22:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:22:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:22:12,401 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:22:12,510 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:22:12,526 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:22:12,905 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:22:13,343 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:22:13,440 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:22:13,552 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
namespacegcworker stdout | 2025-02-07 03:22:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:22:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:31.621131+00:00 (in 17.996399 seconds)
namespacegcworker stdout | 2025-02-07 03:22:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:27:13 UTC)" (scheduled at 2025-02-07 03:22:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:22:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 22, 13, 624976), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:22:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 22, 13, 624976), True, datetime.datetime(2025, 2, 7, 3, 22, 13, 624976), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:22:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 22, 13, 624976), True, datetime.datetime(2025, 2, 7, 3, 22, 13, 624976), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 22, 13, 624976), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:22:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:22:13,634 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:27:13 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:22:14,235 [238] [DEBUG] [app] Starting request: urn:request:f83789fb-d4ad-45ec-9375-df5dadfafb59 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:14,235 [235] [DEBUG] [app] Starting request: urn:request:46cd30f8-b5e1-4a08-8191-89f64cd6bd27 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:22:14,239 [245] [DEBUG] [app] Starting request: urn:request:74c918e0-200c-436d-96fb-9a53f3c85156 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:14,239 [243] [DEBUG] [app] Starting request: urn:request:47ddcf6c-c006-43f8-85ea-38db461ed73c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:14,240 [245] [DEBUG] [app] Ending request: urn:request:74c918e0-200c-436d-96fb-9a53f3c85156 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:74c918e0-200c-436d-96fb-9a53f3c85156', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:22:14,240 [243] [DEBUG] [app] Ending request: urn:request:47ddcf6c-c006-43f8-85ea-38db461ed73c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:47ddcf6c-c006-43f8-85ea-38db461ed73c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:22:14,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:22:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:14,243 [236] [DEBUG] [app] Starting request: urn:request:0434e5ba-007b-4eb7-a23b-10fd8c060de8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:22:14,243 [237] [DEBUG] [app] Starting request: urn:request:517113b9-c1aa-4b25-878f-83cea4069ea3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:22:14,244 [236] [DEBUG] [app] Ending request: urn:request:0434e5ba-007b-4eb7-a23b-10fd8c060de8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0434e5ba-007b-4eb7-a23b-10fd8c060de8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:14,244 [237] [DEBUG] [app] Ending request: urn:request:517113b9-c1aa-4b25-878f-83cea4069ea3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:517113b9-c1aa-4b25-878f-83cea4069ea3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:22:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:22:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:22:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:14,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:14,256 [238] [DEBUG] [app] Ending request: urn:request:f83789fb-d4ad-45ec-9375-df5dadfafb59 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f83789fb-d4ad-45ec-9375-df5dadfafb59', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:14,256 [235] [DEBUG] [app] Ending request: urn:request:46cd30f8-b5e1-4a08-8191-89f64cd6bd27 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:46cd30f8-b5e1-4a08-8191-89f64cd6bd27', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:22:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:22:14,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:22:14,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.023)
namespacegcworker stdout | 2025-02-07 03:22:15,032 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:22:15,494 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:22:15,636 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:22:15,639 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:22:15,642 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:22:15,645 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:22:15,648 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:22:15,774 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:22:16,669 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:22:16,672 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:22:16,675 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:22:17,578 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:22:17,583 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:22:17,585 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:22:17,587 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:22:17,595 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:22:17,598 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:22:17,602 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:22:17,605 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:22:17,609 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:22:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:28.732147+00:00 (in 9.999530 seconds)
notificationworker stdout | 2025-02-07 03:22:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:28 UTC)" (scheduled at 2025-02-07 03:22:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:22:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:22:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 18, 732917), True, datetime.datetime(2025, 2, 7, 3, 22, 18, 732917), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:22:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:22:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:22:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:22:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:22:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:03.419110+00:00 (in 39.997746 seconds)
repositorygcworker stdout | 2025-02-07 03:22:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:23 UTC)" (scheduled at 2025-02-07 03:22:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:22:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:22:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:22:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:22:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:40.424250+00:00 (in 16.000731 seconds)
gcworker stdout | 2025-02-07 03:22:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:54 UTC)" (scheduled at 2025-02-07 03:22:24.423055+00:00)
gcworker stdout | 2025-02-07 03:22:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:22:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:22:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:22:24,756 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:22:26,675 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:22:27,720 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:22:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:33.814166+00:00 (in 5.081491 seconds)
notificationworker stdout | 2025-02-07 03:22:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:38 UTC)" (scheduled at 2025-02-07 03:22:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:22:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:22:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 28, 733003), True, datetime.datetime(2025, 2, 7, 3, 22, 28, 733003), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:22:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:22:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:22:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:22:29,235 [238] [DEBUG] [app] Starting request: urn:request:f689bbf3-574a-4499-b5c5-ff39ad0cb837 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:29,235 [235] [DEBUG] [app] Starting request: urn:request:63a84e82-3135-4729-9a7c-8007934f93aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:22:29,239 [245] [DEBUG] [app] Starting request: urn:request:b9d935f1-5600-4219-b009-d5ecb45c04f1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:29,239 [246] [DEBUG] [app] Starting request: urn:request:5aa1db48-6a8d-460f-9151-f955e8b9cc3b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:29,240 [245] [DEBUG] [app] Ending request: urn:request:b9d935f1-5600-4219-b009-d5ecb45c04f1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9d935f1-5600-4219-b009-d5ecb45c04f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:22:29,240 [246] [DEBUG] [app] Ending request: urn:request:5aa1db48-6a8d-460f-9151-f955e8b9cc3b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5aa1db48-6a8d-460f-9151-f955e8b9cc3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:22:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:22:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:22:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:29,244 [236] [DEBUG] [app] Starting request: urn:request:b8078d66-458d-4fff-81e2-1e3376ed1915 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:22:29,244 [237] [DEBUG] [app] Starting request: urn:request:c0698aa9-f93d-44d8-89c9-f8c863e0b2ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:22:29,244 [236] [DEBUG] [app] Ending request: urn:request:b8078d66-458d-4fff-81e2-1e3376ed1915 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8078d66-458d-4fff-81e2-1e3376ed1915', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:29,244 [237] [DEBUG] [app] Ending request: urn:request:c0698aa9-f93d-44d8-89c9-f8c863e0b2ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c0698aa9-f93d-44d8-89c9-f8c863e0b2ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:22:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:29,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:22:29,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:22:29,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:29,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:29,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:29,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:29,255 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:29,255 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:29,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:29,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:29,258 [235] [DEBUG] [app] Ending request: urn:request:63a84e82-3135-4729-9a7c-8007934f93aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63a84e82-3135-4729-9a7c-8007934f93aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:29,258 [238] [DEBUG] [app] Ending request: urn:request:f689bbf3-574a-4499-b5c5-ff39ad0cb837 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f689bbf3-574a-4499-b5c5-ff39ad0cb837', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:29,258 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:22:29,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:22:29,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:22:29,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
namespacegcworker stdout | 2025-02-07 03:22:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:22:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:32.625945+00:00 (in 1.004388 seconds)
namespacegcworker stdout | 2025-02-07 03:22:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:31 UTC)" (scheduled at 2025-02-07 03:22:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:22:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:22:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 31, 621834), True, datetime.datetime(2025, 2, 7, 3, 22, 31, 621834), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:22:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:22:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:22:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:22:31,997 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:22:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:22:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:31.621131+00:00 (in 58.994721 seconds)
namespacegcworker stdout | 2025-02-07 03:22:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:32 UTC)" (scheduled at 2025-02-07 03:22:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:22:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:22:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:22:33,037 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 03:22:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:38.732147+00:00 (in 4.917420 seconds)
notificationworker stdout | 2025-02-07 03:22:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:27:33 UTC)" (scheduled at 2025-02-07 03:22:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:22:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 22, 33, 815024), 'notification/%'])
notificationworker stdout | 2025-02-07 03:22:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 22, 33, 815024), True, datetime.datetime(2025, 2, 7, 3, 22, 33, 815024), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:22:33,824 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 22, 33, 815024), True, datetime.datetime(2025, 2, 7, 3, 22, 33, 815024), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 22, 33, 815024), 'notification/%'])
notificationworker stdout | 2025-02-07 03:22:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:22:33,826 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:27:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:22:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:22:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:03.919286+00:00 (in 29.999545 seconds)
buildlogsarchiver stdout | 2025-02-07 03:22:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:03 UTC)" (scheduled at 2025-02-07 03:22:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:22:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 22, 33, 919997), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:22:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:22:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:22:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:03 UTC)" executed successfully
builder stdout | 2025-02-07 03:22:37,688 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:56.713547+00:00 (in 18.997238 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:37 UTC)" (scheduled at 2025-02-07 03:22:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:37 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:22:38,129 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:22:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:48.732147+00:00 (in 9.999338 seconds)
notificationworker stdout | 2025-02-07 03:22:38,733 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:48 UTC)" (scheduled at 2025-02-07 03:22:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:22:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:22:38,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 38, 733274), True, datetime.datetime(2025, 2, 7, 3, 22, 38, 733274), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:22:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:22:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:22:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:22:39,424 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:22:40,230 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:22:40,319 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:22:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:22:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:54.423055+00:00 (in 13.998357 seconds)
gcworker stdout | 2025-02-07 03:22:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:10 UTC)" (scheduled at 2025-02-07 03:22:40.424250+00:00)
gcworker stdout | 2025-02-07 03:22:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:22:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898260431, None, 1, 0])
gcworker stdout | 2025-02-07 03:22:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:22:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:22:41,398 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:41,653 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:22:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:22:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:11.831521+00:00 (in 29.999505 seconds)
autopruneworker stdout | 2025-02-07 03:22:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:11 UTC)" (scheduled at 2025-02-07 03:22:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:22:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894961836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:22:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:22:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:22:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:22:42,437 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:22:42,523 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:22:42,562 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:22:42,918 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:22:43,379 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:22:43,476 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:22:43,575 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:22:44,235 [235] [DEBUG] [app] Starting request: urn:request:bb79c7ba-c65e-452c-ab0c-6913736dfa6e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:44,235 [238] [DEBUG] [app] Starting request: urn:request:8b639538-b346-48ae-8e7e-816a5919afb9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:22:44,238 [244] [DEBUG] [app] Starting request: urn:request:9294089f-b226-46cf-a033-32a8b0d4c4e5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:44,238 [239] [DEBUG] [app] Starting request: urn:request:b95487d1-f8af-462d-95aa-b99f94be5b00 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:44,239 [244] [DEBUG] [app] Ending request: urn:request:9294089f-b226-46cf-a033-32a8b0d4c4e5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9294089f-b226-46cf-a033-32a8b0d4c4e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:22:44,239 [239] [DEBUG] [app] Ending request: urn:request:b95487d1-f8af-462d-95aa-b99f94be5b00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b95487d1-f8af-462d-95aa-b99f94be5b00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:22:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:22:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:44,242 [235] [DEBUG] [app] Starting request: urn:request:1eac564d-6220-4d66-88d1-dad09ce47be1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:22:44,242 [235] [DEBUG] [app] Ending request: urn:request:1eac564d-6220-4d66-88d1-dad09ce47be1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1eac564d-6220-4d66-88d1-dad09ce47be1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:44,243 [238] [DEBUG] [app] Starting request: urn:request:56f784fe-5d14-44fa-a74e-7d78ad7d504a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:22:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:44,243 [238] [DEBUG] [app] Ending request: urn:request:56f784fe-5d14-44fa-a74e-7d78ad7d504a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56f784fe-5d14-44fa-a74e-7d78ad7d504a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:22:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:22:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:44,255 [235] [DEBUG] [app] Ending request: urn:request:bb79c7ba-c65e-452c-ab0c-6913736dfa6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb79c7ba-c65e-452c-ab0c-6913736dfa6e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
gunicorn-web stdout | 2025-02-07 03:22:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:22:44,256 [238] [DEBUG] [app] Ending request: urn:request:8b639538-b346-48ae-8e7e-816a5919afb9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b639538-b346-48ae-8e7e-816a5919afb9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:22:44,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
namespacegcworker stdout | 2025-02-07 03:22:45,068 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:22:45,531 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:22:45,644 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:22:45,647 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:22:45,650 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:22:45,652 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:22:45,655 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:22:45,810 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:22:46,676 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:22:46,679 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:22:46,681 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:22:47,589 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:22:47,593 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:22:47,596 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:22:47,598 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:22:47,605 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:22:47,607 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:22:47,611 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:22:47,614 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:22:47,618 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:22:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:50.814677+00:00 (in 2.081998 seconds)
notificationworker stdout | 2025-02-07 03:22:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:58 UTC)" (scheduled at 2025-02-07 03:22:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:22:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:22:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 48, 732960), True, datetime.datetime(2025, 2, 7, 3, 22, 48, 732960), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:22:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:22:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:22:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:22:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:22:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:22:58.732147+00:00 (in 7.917016 seconds)
notificationworker stdout | 2025-02-07 03:22:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:50 UTC)" (scheduled at 2025-02-07 03:22:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:22:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:22:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:22:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:22:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:10.424250+00:00 (in 16.000690 seconds)
gcworker stdout | 2025-02-07 03:22:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:24 UTC)" (scheduled at 2025-02-07 03:22:54.423055+00:00)
gcworker stdout | 2025-02-07 03:22:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:22:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:22:54,782 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:22:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:22:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:56.118855+00:00 (in 59.999524 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:22:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:56 UTC)" (scheduled at 2025-02-07 03:22:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:22:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:22:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:22:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:22:56,712 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:37.715878+00:00 (in 41.001848 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:56 UTC)" (scheduled at 2025-02-07 03:22:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:22:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:56 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:22:57,730 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:22:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:22:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:08.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 03:22:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:08 UTC)" (scheduled at 2025-02-07 03:22:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:22:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:22:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 22, 58, 732889), True, datetime.datetime(2025, 2, 7, 3, 22, 58, 732889), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:22:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:22:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:22:58,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:22:59,235 [238] [DEBUG] [app] Starting request: urn:request:fa26be48-958f-41fe-9e96-6f69c617d4a7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:59,235 [236] [DEBUG] [app] Starting request: urn:request:783afa5b-f742-48f1-9346-8005fab7621d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:22:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:22:59,239 [246] [DEBUG] [app] Starting request: urn:request:245c3a99-32ed-4b3a-bf0b-49185f41124c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:59,239 [239] [DEBUG] [app] Starting request: urn:request:88f42e36-16e4-4ddf-a94a-bee1c2335c51 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:22:59,239 [239] [DEBUG] [app] Ending request: urn:request:88f42e36-16e4-4ddf-a94a-bee1c2335c51 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:88f42e36-16e4-4ddf-a94a-bee1c2335c51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:22:59,239 [246] [DEBUG] [app] Ending request: urn:request:245c3a99-32ed-4b3a-bf0b-49185f41124c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:245c3a99-32ed-4b3a-bf0b-49185f41124c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:22:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:22:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 03:22:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:22:59,242 [238] [DEBUG] [app] Starting request: urn:request:eec16a18-134b-404e-8fcb-5906ac546756 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:22:59,242 [235] [DEBUG] [app] Starting request: urn:request:a41dee9f-e1a0-4245-b3be-4fb2e3cdd0a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:22:59,242 [238] [DEBUG] [app] Ending request: urn:request:eec16a18-134b-404e-8fcb-5906ac546756 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eec16a18-134b-404e-8fcb-5906ac546756', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:59,242 [235] [DEBUG] [app] Ending request: urn:request:a41dee9f-e1a0-4245-b3be-4fb2e3cdd0a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a41dee9f-e1a0-4245-b3be-4fb2e3cdd0a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:22:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:22:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:22:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:22:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:22:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:22:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:22:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:22:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:22:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:22:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:22:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:22:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:22:59,254 [236] [DEBUG] [app] Ending request: urn:request:783afa5b-f742-48f1-9346-8005fab7621d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:783afa5b-f742-48f1-9346-8005fab7621d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:59,255 [238] [DEBUG] [app] Ending request: urn:request:fa26be48-958f-41fe-9e96-6f69c617d4a7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fa26be48-958f-41fe-9e96-6f69c617d4a7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:22:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:22:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:22:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:22:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:22:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:22:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:22:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:22:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:04.936688+00:00 (in 4.998274 seconds)
exportactionlogsworker stdout | 2025-02-07 03:22:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:59 UTC)" (scheduled at 2025-02-07 03:22:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:22:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:22:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:23:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:23:02,030 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:23:03,044 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:23:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:23:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:23.420886+00:00 (in 20.001270 seconds)
repositorygcworker stdout | 2025-02-07 03:23:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:03 UTC)" (scheduled at 2025-02-07 03:23:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:23:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:23:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 3, 419934), True, datetime.datetime(2025, 2, 7, 3, 23, 3, 419934), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:23:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:23:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:23:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:23:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:23:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:33.919286+00:00 (in 29.999537 seconds)
buildlogsarchiver stdout | 2025-02-07 03:23:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:33 UTC)" (scheduled at 2025-02-07 03:23:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:23:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 23, 3, 920023), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:23:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:23:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:23:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:23:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:23:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:59.937899+00:00 (in 55.000711 seconds)
exportactionlogsworker stdout | 2025-02-07 03:23:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:04 UTC)" (scheduled at 2025-02-07 03:23:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:23:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:23:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 4, 937538), True, datetime.datetime(2025, 2, 7, 3, 23, 4, 937538), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:23:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:23:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:23:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:23:07,724 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:23:08,165 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:23:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:23:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:18.732147+00:00 (in 9.999528 seconds)
notificationworker stdout | 2025-02-07 03:23:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:18 UTC)" (scheduled at 2025-02-07 03:23:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:23:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:23:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 8, 732915), True, datetime.datetime(2025, 2, 7, 3, 23, 8, 732915), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:23:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:23:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:23:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:23:09,435 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:23:10,259 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:23:10,329 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:23:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:23:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:24.423055+00:00 (in 13.998262 seconds)
gcworker stdout | 2025-02-07 03:23:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:40 UTC)" (scheduled at 2025-02-07 03:23:10.424250+00:00)
gcworker stdout | 2025-02-07 03:23:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:23:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898290431, None, 1, 0])
gcworker stdout | 2025-02-07 03:23:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:23:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:23:11,432 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:11,689 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:23:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:23:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:41.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 03:23:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:41 UTC)" (scheduled at 2025-02-07 03:23:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:23:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738894991836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:23:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:23:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:23:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:23:12,474 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:23:12,546 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:23:12,597 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:23:12,955 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:23:13,415 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:23:13,506 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:23:13,592 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:23:14,235 [235] [DEBUG] [app] Starting request: urn:request:b893df8c-2a74-4c9c-8ddb-41260f1035f6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:14,235 [237] [DEBUG] [app] Starting request: urn:request:cb4d8529-4cc9-442f-b3dc-c849b94be3ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:23:14,238 [245] [DEBUG] [app] Starting request: urn:request:ee068f3e-da45-4588-b123-41ddb8fb34c5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:14,238 [239] [DEBUG] [app] Starting request: urn:request:23290518-9d61-4689-a9ce-cde07eba4e4a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:14,239 [239] [DEBUG] [app] Ending request: urn:request:23290518-9d61-4689-a9ce-cde07eba4e4a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23290518-9d61-4689-a9ce-cde07eba4e4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:23:14,239 [245] [DEBUG] [app] Ending request: urn:request:ee068f3e-da45-4588-b123-41ddb8fb34c5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee068f3e-da45-4588-b123-41ddb8fb34c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:23:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:23:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:23:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:14,242 [235] [DEBUG] [app] Starting request: urn:request:a96d19d4-99d5-4851-bfba-d7095c8205cb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:14,242 [237] [DEBUG] [app] Starting request: urn:request:ec7207c9-0d5a-42dd-a513-b164c034c371 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:14,242 [235] [DEBUG] [app] Ending request: urn:request:a96d19d4-99d5-4851-bfba-d7095c8205cb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a96d19d4-99d5-4851-bfba-d7095c8205cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:14,242 [237] [DEBUG] [app] Ending request: urn:request:ec7207c9-0d5a-42dd-a513-b164c034c371 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec7207c9-0d5a-42dd-a513-b164c034c371', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:23:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:23:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:14,254 [237] [DEBUG] [app] Ending request: urn:request:cb4d8529-4cc9-442f-b3dc-c849b94be3ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb4d8529-4cc9-442f-b3dc-c849b94be3ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:23:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:23:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:23:14,255 [235] [DEBUG] [app] Ending request: urn:request:b893df8c-2a74-4c9c-8ddb-41260f1035f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b893df8c-2a74-4c9c-8ddb-41260f1035f6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 03:23:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:23:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:23:15,106 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:23:15,547 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:23:15,654 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:23:15,658 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:23:15,661 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:23:15,664 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:23:15,667 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:23:15,818 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:23:16,684 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:23:16,687 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:23:16,689 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:23:17,597 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:23:17,602 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:23:17,604 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:23:17,607 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:23:17,615 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:23:17,618 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:23:17,621 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:23:17,624 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:23:17,626 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:23:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:23:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:28.732147+00:00 (in 9.999528 seconds)
notificationworker stdout | 2025-02-07 03:23:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:28 UTC)" (scheduled at 2025-02-07 03:23:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:23:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:23:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 18, 732918), True, datetime.datetime(2025, 2, 7, 3, 23, 18, 732918), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:23:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:23:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:23:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:23:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:23:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:32.420707+00:00 (in 8.999273 seconds)
repositorygcworker stdout | 2025-02-07 03:23:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:23 UTC)" (scheduled at 2025-02-07 03:23:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:23:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:23:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:23:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:23:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 03:23:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:54 UTC)" (scheduled at 2025-02-07 03:23:24.423055+00:00)
gcworker stdout | 2025-02-07 03:23:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:23:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:23:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:23:24,802 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:23:26,747 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:23:27,752 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:23:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:23:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:38.732147+00:00 (in 9.999518 seconds)
notificationworker stdout | 2025-02-07 03:23:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:38 UTC)" (scheduled at 2025-02-07 03:23:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:23:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:23:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 28, 732935), True, datetime.datetime(2025, 2, 7, 3, 23, 28, 732935), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:23:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:23:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:23:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:23:29,235 [238] [DEBUG] [app] Starting request: urn:request:373721bf-30fb-4411-ba29-0064f5a19d09 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:29,235 [237] [DEBUG] [app] Starting request: urn:request:15ff1321-2266-497d-8fa5-da1065b35aa5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:23:29,239 [246] [DEBUG] [app] Starting request: urn:request:33f3e99a-a6b6-4371-bef3-0a023968b7e4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:29,239 [239] [DEBUG] [app] Starting request: urn:request:3f619d2f-261f-43b8-b184-f53b70c345c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:29,239 [246] [DEBUG] [app] Ending request: urn:request:33f3e99a-a6b6-4371-bef3-0a023968b7e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:33f3e99a-a6b6-4371-bef3-0a023968b7e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:23:29,239 [239] [DEBUG] [app] Ending request: urn:request:3f619d2f-261f-43b8-b184-f53b70c345c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f619d2f-261f-43b8-b184-f53b70c345c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:23:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:23:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:23:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:29,243 [235] [DEBUG] [app] Starting request: urn:request:9e346388-6d36-422d-8b72-77e54e5a6a69 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:29,243 [235] [DEBUG] [app] Ending request: urn:request:9e346388-6d36-422d-8b72-77e54e5a6a69 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9e346388-6d36-422d-8b72-77e54e5a6a69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:29,243 [236] [DEBUG] [app] Starting request: urn:request:a1570853-63f1-4e71-a150-c640726d2f65 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:23:29,244 [236] [DEBUG] [app] Ending request: urn:request:a1570853-63f1-4e71-a150-c640726d2f65 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1570853-63f1-4e71-a150-c640726d2f65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:23:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:23:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:29,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:29,255 [238] [DEBUG] [app] Ending request: urn:request:373721bf-30fb-4411-ba29-0064f5a19d09 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:373721bf-30fb-4411-ba29-0064f5a19d09', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:23:29,255 [237] [DEBUG] [app] Ending request: urn:request:15ff1321-2266-497d-8fa5-da1065b35aa5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15ff1321-2266-497d-8fa5-da1065b35aa5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:23:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:23:29,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:23:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:23:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
namespacegcworker stdout | 2025-02-07 03:23:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:23:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:32.625945+00:00 (in 1.004332 seconds)
namespacegcworker stdout | 2025-02-07 03:23:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:31 UTC)" (scheduled at 2025-02-07 03:23:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:23:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:23:31,623 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 31, 621930), True, datetime.datetime(2025, 2, 7, 3, 23, 31, 621930), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:23:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:23:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:23:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:23:32,037 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:23:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:23:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:03.419110+00:00 (in 30.997936 seconds)
repositorygcworker stdout | 2025-02-07 03:23:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:28:32 UTC)" (scheduled at 2025-02-07 03:23:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:23:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 23, 32, 421453), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:23:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 23, 32, 421453), True, datetime.datetime(2025, 2, 7, 3, 23, 32, 421453), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:23:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 23, 32, 421453), True, datetime.datetime(2025, 2, 7, 3, 23, 32, 421453), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 23, 32, 421453), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:23:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:23:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:28:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:23:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:23:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:31.621131+00:00 (in 58.994711 seconds)
namespacegcworker stdout | 2025-02-07 03:23:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:32 UTC)" (scheduled at 2025-02-07 03:23:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:23:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:23:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:23:33,079 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:23:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:23:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:03.919286+00:00 (in 29.999509 seconds)
buildlogsarchiver stdout | 2025-02-07 03:23:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:03 UTC)" (scheduled at 2025-02-07 03:23:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:23:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 23, 33, 920072), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:23:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:23:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:23:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:56.713547+00:00 (in 18.997122 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:37 UTC)" (scheduled at 2025-02-07 03:23:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:37,724 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:23:37,735 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:23:38,201 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:23:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:23:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:48.732147+00:00 (in 9.999504 seconds)
notificationworker stdout | 2025-02-07 03:23:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:48 UTC)" (scheduled at 2025-02-07 03:23:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:23:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:23:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 38, 732979), True, datetime.datetime(2025, 2, 7, 3, 23, 38, 732979), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:23:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:23:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:23:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:23:39,472 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:23:40,279 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:23:40,366 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:23:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:23:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:54.423055+00:00 (in 13.998366 seconds)
gcworker stdout | 2025-02-07 03:23:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:10 UTC)" (scheduled at 2025-02-07 03:23:40.424250+00:00)
gcworker stdout | 2025-02-07 03:23:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:23:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898320431, None, 1, 0])
gcworker stdout | 2025-02-07 03:23:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:23:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:23:41,458 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:41,725 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:23:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:23:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:11.831521+00:00 (in 29.999534 seconds)
autopruneworker stdout | 2025-02-07 03:23:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:11 UTC)" (scheduled at 2025-02-07 03:23:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:23:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895021836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:23:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:23:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:23:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:23:42,510 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:23:42,579 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:23:42,603 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:23:42,978 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:23:43,445 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:23:43,538 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:23:43,610 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:23:44,236 [238] [DEBUG] [app] Starting request: urn:request:7dc1c837-4f4c-4fbb-aafb-5c186a88ebf5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:44,236 [237] [DEBUG] [app] Starting request: urn:request:c099deac-ec15-456c-9814-50a8dfed1db4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:23:44,239 [246] [DEBUG] [app] Starting request: urn:request:be835c4b-9528-43b4-a749-f9fe4b7aef4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:44,239 [239] [DEBUG] [app] Starting request: urn:request:d5323356-316e-41b7-b176-c510168547c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:44,239 [239] [DEBUG] [app] Ending request: urn:request:d5323356-316e-41b7-b176-c510168547c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d5323356-316e-41b7-b176-c510168547c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:23:44,239 [246] [DEBUG] [app] Ending request: urn:request:be835c4b-9528-43b4-a749-f9fe4b7aef4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be835c4b-9528-43b4-a749-f9fe4b7aef4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:23:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:23:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:23:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:44,242 [237] [DEBUG] [app] Starting request: urn:request:e5e2159c-2f83-43d7-a7b5-543f6a57920d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:44,242 [236] [DEBUG] [app] Starting request: urn:request:397ba5c8-671e-4a18-b602-cb8a76a8c038 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:44,242 [237] [DEBUG] [app] Ending request: urn:request:e5e2159c-2f83-43d7-a7b5-543f6a57920d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5e2159c-2f83-43d7-a7b5-543f6a57920d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:44,242 [236] [DEBUG] [app] Ending request: urn:request:397ba5c8-671e-4a18-b602-cb8a76a8c038 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:397ba5c8-671e-4a18-b602-cb8a76a8c038', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:23:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:23:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:44,254 [237] [DEBUG] [app] Ending request: urn:request:c099deac-ec15-456c-9814-50a8dfed1db4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c099deac-ec15-456c-9814-50a8dfed1db4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:23:44,254 [238] [DEBUG] [app] Ending request: urn:request:7dc1c837-4f4c-4fbb-aafb-5c186a88ebf5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7dc1c837-4f4c-4fbb-aafb-5c186a88ebf5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:23:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:23:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:23:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:23:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 03:23:45,142 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:23:45,579 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:23:45,662 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:23:45,665 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:23:45,669 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:23:45,672 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:23:45,675 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:23:45,854 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:23:46,692 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:23:46,696 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:23:46,698 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:23:47,606 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:23:47,609 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:23:47,612 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:23:47,616 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:23:47,624 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:23:47,627 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:23:47,630 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:23:47,634 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:23:47,637 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:23:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:23:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:50.814677+00:00 (in 2.082078 seconds)
notificationworker stdout | 2025-02-07 03:23:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:58 UTC)" (scheduled at 2025-02-07 03:23:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:23:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:23:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 48, 732874), True, datetime.datetime(2025, 2, 7, 3, 23, 48, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:23:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:23:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:23:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:23:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:23:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:23:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:23:58.732147+00:00 (in 7.917002 seconds)
notificationworker stdout | 2025-02-07 03:23:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:50 UTC)" (scheduled at 2025-02-07 03:23:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:23:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:23:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:23:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:23:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:10.424250+00:00 (in 16.000718 seconds)
gcworker stdout | 2025-02-07 03:23:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:24 UTC)" (scheduled at 2025-02-07 03:23:54.423055+00:00)
gcworker stdout | 2025-02-07 03:23:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:23:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:23:54,838 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:23:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:23:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:56.118855+00:00 (in 59.999481 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:23:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:56 UTC)" (scheduled at 2025-02-07 03:23:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:23:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:23:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:23:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:37.715878+00:00 (in 41.001868 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:56 UTC)" (scheduled at 2025-02-07 03:23:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:23:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:23:56,783 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:23:57,789 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:23:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:23:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:08.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 03:23:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:08 UTC)" (scheduled at 2025-02-07 03:23:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:23:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:23:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 23, 58, 732886), True, datetime.datetime(2025, 2, 7, 3, 23, 58, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:23:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:23:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:23:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:23:59,235 [235] [DEBUG] [app] Starting request: urn:request:71183ac3-5ab3-40c3-b12d-4d3eeb7113a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:59,235 [238] [DEBUG] [app] Starting request: urn:request:ba4e3160-9125-43f6-ae0d-e31ced5b336e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:23:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:23:59,238 [246] [DEBUG] [app] Starting request: urn:request:8c637a31-aa2a-491f-8328-195c5949ff20 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:59,238 [239] [DEBUG] [app] Starting request: urn:request:f04283f0-fa91-4d11-b1e5-d0f208858d5e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:23:59,238 [246] [DEBUG] [app] Ending request: urn:request:8c637a31-aa2a-491f-8328-195c5949ff20 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c637a31-aa2a-491f-8328-195c5949ff20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:23:59,238 [239] [DEBUG] [app] Ending request: urn:request:f04283f0-fa91-4d11-b1e5-d0f208858d5e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f04283f0-fa91-4d11-b1e5-d0f208858d5e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:23:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:23:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:23:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:23:59,241 [235] [DEBUG] [app] Starting request: urn:request:bcecb40c-1b61-4c21-852a-ec8ac3b78e77 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:59,241 [235] [DEBUG] [app] Ending request: urn:request:bcecb40c-1b61-4c21-852a-ec8ac3b78e77 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bcecb40c-1b61-4c21-852a-ec8ac3b78e77', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:59,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:23:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:59,242 [236] [DEBUG] [app] Starting request: urn:request:e52cb534-2323-4e55-9e40-a72eb474ea96 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:23:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:59,242 [236] [DEBUG] [app] Ending request: urn:request:e52cb534-2323-4e55-9e40-a72eb474ea96 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e52cb534-2323-4e55-9e40-a72eb474ea96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:23:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:23:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:23:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:23:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:23:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:23:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:23:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:23:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:23:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:23:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:23:59,253 [235] [DEBUG] [app] Ending request: urn:request:71183ac3-5ab3-40c3-b12d-4d3eeb7113a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71183ac3-5ab3-40c3-b12d-4d3eeb7113a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:23:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:23:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:23:59,254 [238] [DEBUG] [app] Ending request: urn:request:ba4e3160-9125-43f6-ae0d-e31ced5b336e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba4e3160-9125-43f6-ae0d-e31ced5b336e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:23:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:23:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:23:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:23:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:23:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:23:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:04.936688+00:00 (in 4.998218 seconds)
exportactionlogsworker stdout | 2025-02-07 03:23:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:59 UTC)" (scheduled at 2025-02-07 03:23:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:23:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:23:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:24:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:24:02,058 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:24:03,116 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:24:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:24:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:23.420886+00:00 (in 20.001312 seconds)
repositorygcworker stdout | 2025-02-07 03:24:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:03 UTC)" (scheduled at 2025-02-07 03:24:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:24:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:24:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 3, 419849), True, datetime.datetime(2025, 2, 7, 3, 24, 3, 419849), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:24:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:24:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:24:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:24:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:24:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:33.919286+00:00 (in 29.999484 seconds)
buildlogsarchiver stdout | 2025-02-07 03:24:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:33 UTC)" (scheduled at 2025-02-07 03:24:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:24:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 24, 3, 919993), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:24:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:24:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:24:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:24:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:24:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:52.937648+00:00 (in 48.000524 seconds)
exportactionlogsworker stdout | 2025-02-07 03:24:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:04 UTC)" (scheduled at 2025-02-07 03:24:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:24:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:24:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 4, 937424), True, datetime.datetime(2025, 2, 7, 3, 24, 4, 937424), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:24:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:24:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:24:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:24:07,759 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:24:08,237 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:24:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:24:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:18.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 03:24:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:18 UTC)" (scheduled at 2025-02-07 03:24:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:24:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:24:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 8, 732900), True, datetime.datetime(2025, 2, 7, 3, 24, 8, 732900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:24:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:24:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:24:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:24:09,508 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:24:10,314 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:24:10,387 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:24:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:24:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:24.423055+00:00 (in 13.998392 seconds)
gcworker stdout | 2025-02-07 03:24:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:40 UTC)" (scheduled at 2025-02-07 03:24:10.424250+00:00)
gcworker stdout | 2025-02-07 03:24:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:24:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898350430, None, 1, 0])
gcworker stdout | 2025-02-07 03:24:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:24:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:24:11,494 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:11,738 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:24:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:24:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:41.831521+00:00 (in 29.999489 seconds)
autopruneworker stdout | 2025-02-07 03:24:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:41 UTC)" (scheduled at 2025-02-07 03:24:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:24:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895051836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:24:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:24:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:24:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:24:12,546 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:24:12,589 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:24:12,640 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:24:13,012 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:24:13,482 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:24:13,572 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:24:13,646 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:24:14,235 [237] [DEBUG] [app] Starting request: urn:request:d3837cbc-1a69-49bf-9772-f65aaa1a9c05 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:14,235 [238] [DEBUG] [app] Starting request: urn:request:8dae9f34-f419-4922-9dd0-26cf5d43532f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:24:14,238 [246] [DEBUG] [app] Starting request: urn:request:bfba4b78-0590-4838-8b1e-fc5620017321 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:24:14,238 [246] [DEBUG] [app] Ending request: urn:request:bfba4b78-0590-4838-8b1e-fc5620017321 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bfba4b78-0590-4838-8b1e-fc5620017321', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:24:14,239 [245] [DEBUG] [app] Starting request: urn:request:ddb996f6-8b95-45be-b75e-05a8c814db2b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:24:14,239 [245] [DEBUG] [app] Ending request: urn:request:ddb996f6-8b95-45be-b75e-05a8c814db2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ddb996f6-8b95-45be-b75e-05a8c814db2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:24:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:14,241 [235] [DEBUG] [app] Starting request: urn:request:f577a944-443a-4693-b93e-5f4e9c4c623e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:14,242 [235] [DEBUG] [app] Ending request: urn:request:f577a944-443a-4693-b93e-5f4e9c4c623e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f577a944-443a-4693-b93e-5f4e9c4c623e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:24:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:24:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:14,242 [235] [DEBUG] [app] Starting request: urn:request:1cdaeeae-cb6f-43b3-9101-9ec02a970941 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:14,243 [235] [DEBUG] [app] Ending request: urn:request:1cdaeeae-cb6f-43b3-9101-9ec02a970941 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1cdaeeae-cb6f-43b3-9101-9ec02a970941', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:24:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:24:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:14,253 [237] [DEBUG] [app] Ending request: urn:request:d3837cbc-1a69-49bf-9772-f65aaa1a9c05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d3837cbc-1a69-49bf-9772-f65aaa1a9c05', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:24:14,254 [238] [DEBUG] [app] Ending request: urn:request:8dae9f34-f419-4922-9dd0-26cf5d43532f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8dae9f34-f419-4922-9dd0-26cf5d43532f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
namespacegcworker stdout | 2025-02-07 03:24:15,178 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:24:15,606 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:24:15,670 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:24:15,674 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:24:15,677 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:24:15,680 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:24:15,683 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:24:15,887 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:24:16,700 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:24:16,703 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:24:16,706 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:24:17,614 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:24:17,617 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:24:17,621 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:24:17,624 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:24:17,632 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:24:17,635 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:24:17,637 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:24:17,640 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:24:17,644 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:24:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:24:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:28.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 03:24:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:28 UTC)" (scheduled at 2025-02-07 03:24:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:24:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:24:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 18, 732899), True, datetime.datetime(2025, 2, 7, 3, 24, 18, 732899), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:24:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:24:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:24:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:24:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:24:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:03.419110+00:00 (in 39.997742 seconds)
repositorygcworker stdout | 2025-02-07 03:24:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:23 UTC)" (scheduled at 2025-02-07 03:24:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:24:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:24:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:24:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:24:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:40.424250+00:00 (in 16.000718 seconds)
gcworker stdout | 2025-02-07 03:24:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:54 UTC)" (scheduled at 2025-02-07 03:24:24.423055+00:00)
gcworker stdout | 2025-02-07 03:24:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:24:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:24:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:24:24,860 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:24:26,819 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:24:27,826 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:24:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:24:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:38.732147+00:00 (in 9.999533 seconds)
notificationworker stdout | 2025-02-07 03:24:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:38 UTC)" (scheduled at 2025-02-07 03:24:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:24:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:24:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 28, 732902), True, datetime.datetime(2025, 2, 7, 3, 24, 28, 732902), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:24:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:24:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:24:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:24:29,236 [237] [DEBUG] [app] Starting request: urn:request:5dc343f9-44b7-4eeb-8d1d-55456a9466aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:29,236 [238] [DEBUG] [app] Starting request: urn:request:83c6efaa-03f2-432c-ae6e-56b388c1c0a0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:24:29,239 [246] [DEBUG] [app] Starting request: urn:request:aa470382-b1aa-4155-9eb7-3e770860fd94 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:24:29,239 [244] [DEBUG] [app] Starting request: urn:request:334c128f-9066-491a-a8d7-56e44681b8ac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:24:29,239 [246] [DEBUG] [app] Ending request: urn:request:aa470382-b1aa-4155-9eb7-3e770860fd94 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa470382-b1aa-4155-9eb7-3e770860fd94', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:29,240 [244] [DEBUG] [app] Ending request: urn:request:334c128f-9066-491a-a8d7-56e44681b8ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:334c128f-9066-491a-a8d7-56e44681b8ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:24:29,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:29,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:29,243 [237] [DEBUG] [app] Starting request: urn:request:7b19e311-0ca9-461d-9f4f-88428c4a7087 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:29,243 [235] [DEBUG] [app] Starting request: urn:request:a7a1ba63-239b-46ef-9518-8294b6d112e6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:29,244 [237] [DEBUG] [app] Ending request: urn:request:7b19e311-0ca9-461d-9f4f-88428c4a7087 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7b19e311-0ca9-461d-9f4f-88428c4a7087', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:24:29,244 [235] [DEBUG] [app] Ending request: urn:request:a7a1ba63-239b-46ef-9518-8294b6d112e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7a1ba63-239b-46ef-9518-8294b6d112e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:24:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:29,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:29,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:29,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:29,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:29,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:29,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:29,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:29,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:29,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:29,256 [237] [DEBUG] [app] Ending request: urn:request:5dc343f9-44b7-4eeb-8d1d-55456a9466aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5dc343f9-44b7-4eeb-8d1d-55456a9466aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:29,256 [238] [DEBUG] [app] Ending request: urn:request:83c6efaa-03f2-432c-ae6e-56b388c1c0a0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83c6efaa-03f2-432c-ae6e-56b388c1c0a0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:29,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:29,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:24:29,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
namespacegcworker stdout | 2025-02-07 03:24:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:24:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:32.625945+00:00 (in 1.004350 seconds)
namespacegcworker stdout | 2025-02-07 03:24:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:31 UTC)" (scheduled at 2025-02-07 03:24:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:24:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:24:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 31, 621882), True, datetime.datetime(2025, 2, 7, 3, 24, 31, 621882), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:24:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:24:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:24:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:24:32,095 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:24:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:24:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:31.621131+00:00 (in 58.994729 seconds)
namespacegcworker stdout | 2025-02-07 03:24:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:32 UTC)" (scheduled at 2025-02-07 03:24:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:24:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:24:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:24:33,152 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:24:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:24:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:03.919286+00:00 (in 29.999527 seconds)
buildlogsarchiver stdout | 2025-02-07 03:24:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:03 UTC)" (scheduled at 2025-02-07 03:24:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:24:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 24, 33, 920034), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:24:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:24:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:24:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:56.713547+00:00 (in 18.997221 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:37 UTC)" (scheduled at 2025-02-07 03:24:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:24:37,796 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:24:38,273 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:24:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:24:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:48.732147+00:00 (in 9.999422 seconds)
notificationworker stdout | 2025-02-07 03:24:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:48 UTC)" (scheduled at 2025-02-07 03:24:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:24:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:24:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 38, 733121), True, datetime.datetime(2025, 2, 7, 3, 24, 38, 733121), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:24:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:24:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:24:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:24:39,544 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:24:40,341 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:24:40,414 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gcworker stdout | 2025-02-07 03:24:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:24:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:54.423055+00:00 (in 13.998363 seconds)
gcworker stdout | 2025-02-07 03:24:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:10 UTC)" (scheduled at 2025-02-07 03:24:40.424250+00:00)
gcworker stdout | 2025-02-07 03:24:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:24:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898380431, None, 1, 0])
gcworker stdout | 2025-02-07 03:24:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:24:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:24:41,531 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:41,774 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:24:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:24:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:11.831521+00:00 (in 29.999551 seconds)
autopruneworker stdout | 2025-02-07 03:24:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:11 UTC)" (scheduled at 2025-02-07 03:24:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:24:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895081836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:24:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:24:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:24:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:11 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:24:42,581 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:24:42,603 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:24:42,676 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:24:43,042 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:24:43,511 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:24:43,602 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:24:43,680 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:24:44,236 [235] [DEBUG] [app] Starting request: urn:request:6eacd20d-1888-4457-8b30-b1321b3581c0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:44,236 [236] [DEBUG] [app] Starting request: urn:request:b399c6d2-f3c2-443e-8f86-1099065e6419 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:24:44,238 [246] [DEBUG] [app] Starting request: urn:request:415d4ea5-491b-4d3a-9247-7bd4147b2def (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:24:44,238 [239] [DEBUG] [app] Starting request: urn:request:a9f0afb0-2668-4aec-8bc4-4dbd2198fe14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:24:44,239 [246] [DEBUG] [app] Ending request: urn:request:415d4ea5-491b-4d3a-9247-7bd4147b2def (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:415d4ea5-491b-4d3a-9247-7bd4147b2def', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:44,239 [239] [DEBUG] [app] Ending request: urn:request:a9f0afb0-2668-4aec-8bc4-4dbd2198fe14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9f0afb0-2668-4aec-8bc4-4dbd2198fe14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:24:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:24:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:44,241 [235] [DEBUG] [app] Starting request: urn:request:434aaebe-b574-4535-99a4-ea687d6e8fe9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:44,242 [235] [DEBUG] [app] Ending request: urn:request:434aaebe-b574-4535-99a4-ea687d6e8fe9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:434aaebe-b574-4535-99a4-ea687d6e8fe9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:24:44,242 [238] [DEBUG] [app] Starting request: urn:request:109f4b62-aee8-439e-a42e-7df84872e41a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:24:44,242 [238] [DEBUG] [app] Ending request: urn:request:109f4b62-aee8-439e-a42e-7df84872e41a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:109f4b62-aee8-439e-a42e-7df84872e41a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:24:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:24:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:44,254 [235] [DEBUG] [app] Ending request: urn:request:6eacd20d-1888-4457-8b30-b1321b3581c0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6eacd20d-1888-4457-8b30-b1321b3581c0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:44,254 [236] [DEBUG] [app] Ending request: urn:request:b399c6d2-f3c2-443e-8f86-1099065e6419 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b399c6d2-f3c2-443e-8f86-1099065e6419', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:24:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 03:24:45,215 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:24:45,642 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:24:45,680 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:24:45,684 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:24:45,687 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:24:45,691 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:24:45,694 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:24:45,925 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:24:46,707 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:24:46,710 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:24:46,713 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:24:47,627 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:24:47,630 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:24:47,633 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:24:47,635 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:24:47,640 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:24:47,643 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:24:47,645 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:24:47,652 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:24:47,655 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:24:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:24:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:50.814677+00:00 (in 2.082060 seconds)
notificationworker stdout | 2025-02-07 03:24:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:58 UTC)" (scheduled at 2025-02-07 03:24:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:24:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:24:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 48, 732896), True, datetime.datetime(2025, 2, 7, 3, 24, 48, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:24:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:24:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:24:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:24:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:24:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:24:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:58.732147+00:00 (in 7.917003 seconds)
notificationworker stdout | 2025-02-07 03:24:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:50 UTC)" (scheduled at 2025-02-07 03:24:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:24:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:24:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:24:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:24:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:24:59.937899+00:00 (in 6.999799 seconds)
exportactionlogsworker stdout | 2025-02-07 03:24:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:29:52 UTC)" (scheduled at 2025-02-07 03:24:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:24:52,939 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 24, 52, 938409), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:24:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 24, 52, 938409), True, datetime.datetime(2025, 2, 7, 3, 24, 52, 938409), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:24:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 24, 52, 938409), True, datetime.datetime(2025, 2, 7, 3, 24, 52, 938409), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 24, 52, 938409), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:24:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:24:52,949 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:29:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:24:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:24:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:10.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 03:24:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:24 UTC)" (scheduled at 2025-02-07 03:24:54.423055+00:00)
gcworker stdout | 2025-02-07 03:24:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:24:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:24:54,896 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:24:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:24:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:56.118855+00:00 (in 59.999471 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:24:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:56 UTC)" (scheduled at 2025-02-07 03:24:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:24:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:24:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:24:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:37.715878+00:00 (in 41.001859 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:56 UTC)" (scheduled at 2025-02-07 03:24:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:24:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:24:56,853 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:24:57,853 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:24:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:24:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:08.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 03:24:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:08 UTC)" (scheduled at 2025-02-07 03:24:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:24:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:24:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 24, 58, 732890), True, datetime.datetime(2025, 2, 7, 3, 24, 58, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:24:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:24:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:24:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:24:59,235 [235] [DEBUG] [app] Starting request: urn:request:9915f64a-ee9b-43a1-a36c-79abf9532d12 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:59,235 [238] [DEBUG] [app] Starting request: urn:request:5693a8bd-fb40-4ac1-a534-24f36fe791f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:24:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:24:59,238 [239] [DEBUG] [app] Starting request: urn:request:06dbf91e-f6f0-4653-a814-a9dad22cdaf6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:24:59,238 [245] [DEBUG] [app] Starting request: urn:request:6f79a26c-0156-4b98-b471-116e6f198206 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:24:59,238 [239] [DEBUG] [app] Ending request: urn:request:06dbf91e-f6f0-4653-a814-a9dad22cdaf6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:06dbf91e-f6f0-4653-a814-a9dad22cdaf6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:59,239 [245] [DEBUG] [app] Ending request: urn:request:6f79a26c-0156-4b98-b471-116e6f198206 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f79a26c-0156-4b98-b471-116e6f198206', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:24:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:24:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:24:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:24:59,241 [235] [DEBUG] [app] Starting request: urn:request:f2582500-00cf-4a32-a704-90954bbe02bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:59,241 [237] [DEBUG] [app] Starting request: urn:request:6d4d0c8d-228c-4be0-aac2-e42a8d6f0839 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:24:59,241 [235] [DEBUG] [app] Ending request: urn:request:f2582500-00cf-4a32-a704-90954bbe02bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2582500-00cf-4a32-a704-90954bbe02bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:24:59,242 [237] [DEBUG] [app] Ending request: urn:request:6d4d0c8d-228c-4be0-aac2-e42a8d6f0839 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d4d0c8d-228c-4be0-aac2-e42a8d6f0839', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:24:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:24:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:24:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:24:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:24:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:24:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:24:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:24:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:24:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:24:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:24:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:24:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:24:59,253 [235] [DEBUG] [app] Ending request: urn:request:9915f64a-ee9b-43a1-a36c-79abf9532d12 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9915f64a-ee9b-43a1-a36c-79abf9532d12', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:59,253 [238] [DEBUG] [app] Ending request: urn:request:5693a8bd-fb40-4ac1-a534-24f36fe791f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5693a8bd-fb40-4ac1-a534-24f36fe791f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:24:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:24:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:24:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:24:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:24:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:24:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:24:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:04.936688+00:00 (in 4.998319 seconds)
exportactionlogsworker stdout | 2025-02-07 03:24:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:59 UTC)" (scheduled at 2025-02-07 03:24:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:24:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:24:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:25:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:25:02,131 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:25:03,162 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:25:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:25:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:23.420886+00:00 (in 20.001313 seconds)
repositorygcworker stdout | 2025-02-07 03:25:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:03 UTC)" (scheduled at 2025-02-07 03:25:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:25:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:25:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 3, 419780), True, datetime.datetime(2025, 2, 7, 3, 25, 3, 419780), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:25:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:25:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:25:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:25:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:25:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:33.919286+00:00 (in 29.999496 seconds)
buildlogsarchiver stdout | 2025-02-07 03:25:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:33 UTC)" (scheduled at 2025-02-07 03:25:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:25:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 25, 3, 920076), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:25:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:25:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:25:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:25:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:25:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:59.937899+00:00 (in 55.000755 seconds)
exportactionlogsworker stdout | 2025-02-07 03:25:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:04 UTC)" (scheduled at 2025-02-07 03:25:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:25:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:25:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 4, 937502), True, datetime.datetime(2025, 2, 7, 3, 25, 4, 937502), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:25:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:25:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:25:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:25:07,831 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:25:08,309 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:25:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:25:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:18.732147+00:00 (in 9.999383 seconds)
notificationworker stdout | 2025-02-07 03:25:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:18 UTC)" (scheduled at 2025-02-07 03:25:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:25:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:25:08,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 8, 733287), True, datetime.datetime(2025, 2, 7, 3, 25, 8, 733287), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:25:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:25:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:25:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:25:09,581 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:25:10,349 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gcworker stdout | 2025-02-07 03:25:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:25:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:24.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 03:25:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:40 UTC)" (scheduled at 2025-02-07 03:25:10.424250+00:00)
gcworker stdout | 2025-02-07 03:25:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:25:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898410431, None, 1, 0])
gcworker stdout | 2025-02-07 03:25:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:25:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:40 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 03:25:10,439 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:25:11,567 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:11,810 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
autopruneworker stdout | 2025-02-07 03:25:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:25:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:41.831521+00:00 (in 29.999537 seconds)
autopruneworker stdout | 2025-02-07 03:25:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:41 UTC)" (scheduled at 2025-02-07 03:25:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:25:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895111836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:25:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:25:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:25:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:41 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:25:12,617 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:25:12,639 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:25:12,698 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:25:13,079 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:25:13,522 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:25:13,629 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:25:13,695 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:25:14,235 [238] [DEBUG] [app] Starting request: urn:request:c5f5e049-e980-49e7-800f-b556160e3810 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:14,235 [235] [DEBUG] [app] Starting request: urn:request:c47d6413-8f9e-4871-a0a1-6cf34ad8bb08 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:25:14,238 [239] [DEBUG] [app] Starting request: urn:request:77b735b7-cde1-4d74-8a3e-a049bc01c56a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:14,238 [246] [DEBUG] [app] Starting request: urn:request:2a813d25-669e-454f-960a-129f1a5dbb52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:14,239 [239] [DEBUG] [app] Ending request: urn:request:77b735b7-cde1-4d74-8a3e-a049bc01c56a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77b735b7-cde1-4d74-8a3e-a049bc01c56a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:25:14,239 [246] [DEBUG] [app] Ending request: urn:request:2a813d25-669e-454f-960a-129f1a5dbb52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a813d25-669e-454f-960a-129f1a5dbb52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:25:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:25:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:14,242 [238] [DEBUG] [app] Starting request: urn:request:bc501e2d-e2e3-4fda-bee8-3d324693ab31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:14,242 [235] [DEBUG] [app] Starting request: urn:request:8ebb2687-fae2-41be-abe8-8d85ddb686c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:14,242 [238] [DEBUG] [app] Ending request: urn:request:bc501e2d-e2e3-4fda-bee8-3d324693ab31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc501e2d-e2e3-4fda-bee8-3d324693ab31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:25:14,242 [235] [DEBUG] [app] Ending request: urn:request:8ebb2687-fae2-41be-abe8-8d85ddb686c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ebb2687-fae2-41be-abe8-8d85ddb686c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:25:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:25:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:25:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:25:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:14,254 [238] [DEBUG] [app] Ending request: urn:request:c5f5e049-e980-49e7-800f-b556160e3810 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c5f5e049-e980-49e7-800f-b556160e3810', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:14,254 [235] [DEBUG] [app] Ending request: urn:request:c47d6413-8f9e-4871-a0a1-6cf34ad8bb08 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c47d6413-8f9e-4871-a0a1-6cf34ad8bb08', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:25:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:25:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:25:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:25:15,251 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:25:15,678 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-web stdout | 2025-02-07 03:25:15,687 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:25:15,691 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:25:15,693 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:25:15,699 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:25:15,701 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
blobuploadcleanupworker stdout | 2025-02-07 03:25:15,961 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:25:16,716 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:25:16,718 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:25:16,721 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:25:17,640 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:25:17,643 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:25:17,646 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:25:17,649 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:25:17,652 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:25:17,655 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:25:17,659 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:25:17,662 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:25:17,665 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:25:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:25:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:28.732147+00:00 (in 9.999527 seconds)
notificationworker stdout | 2025-02-07 03:25:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:28 UTC)" (scheduled at 2025-02-07 03:25:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:25:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:25:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 18, 732897), True, datetime.datetime(2025, 2, 7, 3, 25, 18, 732897), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:25:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:25:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:25:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:25:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:25:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:03.419110+00:00 (in 39.997708 seconds)
repositorygcworker stdout | 2025-02-07 03:25:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:23 UTC)" (scheduled at 2025-02-07 03:25:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:25:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:25:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:25:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:25:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:40.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 03:25:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:54 UTC)" (scheduled at 2025-02-07 03:25:24.423055+00:00)
gcworker stdout | 2025-02-07 03:25:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:25:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:25:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:25:24,932 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:25:26,873 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:25:27,889 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:25:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:25:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:38.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 03:25:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:38 UTC)" (scheduled at 2025-02-07 03:25:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:25:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:25:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 28, 732807), True, datetime.datetime(2025, 2, 7, 3, 25, 28, 732807), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:25:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:25:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:25:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:25:29,236 [237] [DEBUG] [app] Starting request: urn:request:d30445d8-29bc-459c-a075-f0aef56998d4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:29,236 [238] [DEBUG] [app] Starting request: urn:request:db3c6b8e-5fed-4d03-b29f-0ad8053ff5b2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:25:29,238 [239] [DEBUG] [app] Starting request: urn:request:ab219ec3-4158-4a9c-9de0-84192dddda40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:29,239 [239] [DEBUG] [app] Ending request: urn:request:ab219ec3-4158-4a9c-9de0-84192dddda40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab219ec3-4158-4a9c-9de0-84192dddda40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:25:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:25:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:25:29,240 [240] [DEBUG] [app] Starting request: urn:request:f4809433-eae3-406c-bce3-408b28f06458 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:29,241 [240] [DEBUG] [app] Ending request: urn:request:f4809433-eae3-406c-bce3-408b28f06458 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4809433-eae3-406c-bce3-408b28f06458', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:25:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 162 0.004)
gunicorn-registry stdout | 2025-02-07 03:25:29,242 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:29,242 [235] [DEBUG] [app] Starting request: urn:request:593e7b6c-e938-4535-b1bb-6fb53371d4d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:29,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:29,243 [235] [DEBUG] [app] Ending request: urn:request:593e7b6c-e938-4535-b1bb-6fb53371d4d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:593e7b6c-e938-4535-b1bb-6fb53371d4d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:25:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:25:29,244 [235] [DEBUG] [app] Starting request: urn:request:f7205644-a543-4a0c-bd82-beb8811543ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:29,244 [235] [DEBUG] [app] Ending request: urn:request:f7205644-a543-4a0c-bd82-beb8811543ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7205644-a543-4a0c-bd82-beb8811543ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:25:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:25:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:29,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:29,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:29,255 [237] [DEBUG] [app] Ending request: urn:request:d30445d8-29bc-459c-a075-f0aef56998d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d30445d8-29bc-459c-a075-f0aef56998d4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:25:29,255 [238] [DEBUG] [app] Ending request: urn:request:db3c6b8e-5fed-4d03-b29f-0ad8053ff5b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:db3c6b8e-5fed-4d03-b29f-0ad8053ff5b2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:25:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:25:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:25:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:25:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:32.625945+00:00 (in 1.004356 seconds)
namespacegcworker stdout | 2025-02-07 03:25:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:31 UTC)" (scheduled at 2025-02-07 03:25:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:25:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:25:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 31, 621857), True, datetime.datetime(2025, 2, 7, 3, 25, 31, 621857), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:25:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:25:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:25:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:25:32,154 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:25:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:25:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:31.621131+00:00 (in 58.994711 seconds)
namespacegcworker stdout | 2025-02-07 03:25:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:32 UTC)" (scheduled at 2025-02-07 03:25:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:25:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:25:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:25:33,178 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:25:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:25:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:03.919286+00:00 (in 29.999469 seconds)
buildlogsarchiver stdout | 2025-02-07 03:25:33,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:03 UTC)" (scheduled at 2025-02-07 03:25:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:25:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 25, 33, 920112), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:25:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:25:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:25:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:56.713547+00:00 (in 18.997209 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:37 UTC)" (scheduled at 2025-02-07 03:25:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:25:37,867 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:25:38,327 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:25:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:25:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:48.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:25:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:48 UTC)" (scheduled at 2025-02-07 03:25:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:25:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:25:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 38, 732860), True, datetime.datetime(2025, 2, 7, 3, 25, 38, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:25:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:25:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:25:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:25:39,609 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:25:40,375 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gcworker stdout | 2025-02-07 03:25:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:25:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:54.423055+00:00 (in 13.998364 seconds)
gcworker stdout | 2025-02-07 03:25:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:10 UTC)" (scheduled at 2025-02-07 03:25:40.424250+00:00)
gcworker stdout | 2025-02-07 03:25:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:25:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898440431, None, 1, 0])
gcworker stdout | 2025-02-07 03:25:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:25:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:10 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 03:25:40,471 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:25:41,603 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:25:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:25:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:11.831521+00:00 (in 29.999504 seconds)
autopruneworker stdout | 2025-02-07 03:25:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:11 UTC)" (scheduled at 2025-02-07 03:25:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:25:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895141836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:25:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:25:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:25:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:11 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:41,846 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:25:42,629 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:25:42,675 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:25:42,734 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:25:43,089 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:25:43,558 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:25:43,652 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:25:43,717 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:25:44,236 [235] [DEBUG] [app] Starting request: urn:request:0909aa4c-b30e-4faa-a9fb-def7abb2e0ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:44,236 [238] [DEBUG] [app] Starting request: urn:request:cf3136ce-c193-464d-8d4a-619230dd1299 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:25:44,239 [244] [DEBUG] [app] Starting request: urn:request:616b5697-2c82-453b-be4b-781bf560271f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:44,239 [239] [DEBUG] [app] Starting request: urn:request:178d7553-d303-4db3-837a-6275fbe7364e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:44,239 [244] [DEBUG] [app] Ending request: urn:request:616b5697-2c82-453b-be4b-781bf560271f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:616b5697-2c82-453b-be4b-781bf560271f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:25:44,239 [239] [DEBUG] [app] Ending request: urn:request:178d7553-d303-4db3-837a-6275fbe7364e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:178d7553-d303-4db3-837a-6275fbe7364e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:25:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:25:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:44,242 [236] [DEBUG] [app] Starting request: urn:request:28011c06-8f24-4508-8559-6453cccda094 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:44,242 [238] [DEBUG] [app] Starting request: urn:request:a561a893-bebf-44c0-b9eb-422806425163 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:44,242 [236] [DEBUG] [app] Ending request: urn:request:28011c06-8f24-4508-8559-6453cccda094 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28011c06-8f24-4508-8559-6453cccda094', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:25:44,242 [238] [DEBUG] [app] Ending request: urn:request:a561a893-bebf-44c0-b9eb-422806425163 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a561a893-bebf-44c0-b9eb-422806425163', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:25:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:25:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:25:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:25:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:44,254 [238] [DEBUG] [app] Ending request: urn:request:cf3136ce-c193-464d-8d4a-619230dd1299 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf3136ce-c193-464d-8d4a-619230dd1299', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:44,254 [235] [DEBUG] [app] Ending request: urn:request:0909aa4c-b30e-4faa-a9fb-def7abb2e0ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0909aa4c-b30e-4faa-a9fb-def7abb2e0ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:25:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:25:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:25:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:25:45,287 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:25:45,695 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:25:45,699 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:25:45,701 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:25:45,705 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:25:45,707 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:25:45,713 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:25:45,974 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:25:46,723 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:25:46,725 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:25:46,728 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:25:47,651 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:25:47,654 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:25:47,658 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:25:47,661 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:25:47,663 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:25:47,666 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:25:47,668 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:25:47,673 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:25:47,676 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:25:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:25:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:50.814677+00:00 (in 2.082060 seconds)
notificationworker stdout | 2025-02-07 03:25:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:58 UTC)" (scheduled at 2025-02-07 03:25:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:25:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:25:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 48, 732926), True, datetime.datetime(2025, 2, 7, 3, 25, 48, 732926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:25:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:25:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:25:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:25:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:25:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:25:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:25:58.732147+00:00 (in 7.916945 seconds)
notificationworker stdout | 2025-02-07 03:25:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:50 UTC)" (scheduled at 2025-02-07 03:25:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:25:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:25:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:25:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:25:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 03:25:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:24 UTC)" (scheduled at 2025-02-07 03:25:54.423055+00:00)
gcworker stdout | 2025-02-07 03:25:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:25:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:25:54,946 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:25:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:25:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:56.118855+00:00 (in 59.999519 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:25:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:56 UTC)" (scheduled at 2025-02-07 03:25:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:25:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:25:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:25:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:37.715878+00:00 (in 41.001885 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:56 UTC)" (scheduled at 2025-02-07 03:25:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:25:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:25:56,908 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:25:57,924 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:25:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:25:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:08.732147+00:00 (in 9.999427 seconds)
notificationworker stdout | 2025-02-07 03:25:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:08 UTC)" (scheduled at 2025-02-07 03:25:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:25:58,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:25:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 25, 58, 733040), True, datetime.datetime(2025, 2, 7, 3, 25, 58, 733040), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:25:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:25:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:25:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:25:59,235 [238] [DEBUG] [app] Starting request: urn:request:5ab90537-2aa2-4150-be39-252a8f85fad0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:59,235 [235] [DEBUG] [app] Starting request: urn:request:f02af4dc-09b5-4901-bcf4-f48a0c423391 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:25:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:25:59,238 [245] [DEBUG] [app] Starting request: urn:request:abb94118-a66f-4243-9c93-c42bd7bafb48 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:59,238 [246] [DEBUG] [app] Starting request: urn:request:9bd1e060-5daf-421c-8e66-0e2f4993b31c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:25:59,239 [245] [DEBUG] [app] Ending request: urn:request:abb94118-a66f-4243-9c93-c42bd7bafb48 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abb94118-a66f-4243-9c93-c42bd7bafb48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:25:59,239 [246] [DEBUG] [app] Ending request: urn:request:9bd1e060-5daf-421c-8e66-0e2f4993b31c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9bd1e060-5daf-421c-8e66-0e2f4993b31c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:25:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:25:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:25:59,241 [235] [DEBUG] [app] Starting request: urn:request:a5b3cdcc-841e-4c1e-b393-2185797f0a7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:59,241 [235] [DEBUG] [app] Ending request: urn:request:a5b3cdcc-841e-4c1e-b393-2185797f0a7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a5b3cdcc-841e-4c1e-b393-2185797f0a7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:25:59,242 [236] [DEBUG] [app] Starting request: urn:request:2cb99ff2-5a0d-49e8-8c1e-5894399a0df1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:25:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:25:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:59,242 [236] [DEBUG] [app] Ending request: urn:request:2cb99ff2-5a0d-49e8-8c1e-5894399a0df1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2cb99ff2-5a0d-49e8-8c1e-5894399a0df1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:25:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:25:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:25:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:25:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:25:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:25:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:25:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:25:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:25:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:25:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:25:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:25:59,253 [235] [DEBUG] [app] Ending request: urn:request:f02af4dc-09b5-4901-bcf4-f48a0c423391 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f02af4dc-09b5-4901-bcf4-f48a0c423391', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:25:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:25:59,253 [238] [DEBUG] [app] Ending request: urn:request:5ab90537-2aa2-4150-be39-252a8f85fad0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ab90537-2aa2-4150-be39-252a8f85fad0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:25:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:25:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:25:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:25:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:25:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:25:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:04.936688+00:00 (in 4.998305 seconds)
exportactionlogsworker stdout | 2025-02-07 03:25:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:59 UTC)" (scheduled at 2025-02-07 03:25:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:25:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:25:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:26:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:26:02,190 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:26:03,207 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:26:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:26:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:23.420886+00:00 (in 20.001288 seconds)
repositorygcworker stdout | 2025-02-07 03:26:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:03 UTC)" (scheduled at 2025-02-07 03:26:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:26:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:26:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 3, 419933), True, datetime.datetime(2025, 2, 7, 3, 26, 3, 419933), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:26:03,428 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:26:03,428 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:26:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:26:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:26:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:33.919286+00:00 (in 29.999442 seconds)
buildlogsarchiver stdout | 2025-02-07 03:26:03,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:33 UTC)" (scheduled at 2025-02-07 03:26:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:26:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 26, 3, 920135), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:26:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:26:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:26:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:26:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:26:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:59.937899+00:00 (in 55.000761 seconds)
exportactionlogsworker stdout | 2025-02-07 03:26:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:04 UTC)" (scheduled at 2025-02-07 03:26:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:26:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:26:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 4, 937495), True, datetime.datetime(2025, 2, 7, 3, 26, 4, 937495), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:26:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:26:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:26:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:26:07,881 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:26:08,364 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:26:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:26:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:18.732147+00:00 (in 9.999284 seconds)
notificationworker stdout | 2025-02-07 03:26:08,733 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:18 UTC)" (scheduled at 2025-02-07 03:26:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:26:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:26:08,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 8, 733342), True, datetime.datetime(2025, 2, 7, 3, 26, 8, 733342), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:26:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:26:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:26:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:26:09,645 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:26:10,412 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gcworker stdout | 2025-02-07 03:26:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:26:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:24.423055+00:00 (in 13.998328 seconds)
gcworker stdout | 2025-02-07 03:26:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:40 UTC)" (scheduled at 2025-02-07 03:26:10.424250+00:00)
gcworker stdout | 2025-02-07 03:26:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:26:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898470431, None, 1, 0])
gcworker stdout | 2025-02-07 03:26:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:26:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:40 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 03:26:10,507 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:26:11,622 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:26:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:26:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:41.831521+00:00 (in 29.999499 seconds)
autopruneworker stdout | 2025-02-07 03:26:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:41 UTC)" (scheduled at 2025-02-07 03:26:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:26:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895171836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:26:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:26:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:26:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:41 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:11,879 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:26:12,640 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:26:12,711 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:26:12,769 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:26:13,114 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:26:13,578 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:26:13,688 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:26:13,753 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:26:14,235 [236] [DEBUG] [app] Starting request: urn:request:64b9a6e9-9de1-49d0-aa4c-ea6cba212bc1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:14,235 [238] [DEBUG] [app] Starting request: urn:request:a7021f0d-a504-4ac6-8202-78081cb55710 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:26:14,238 [244] [DEBUG] [app] Starting request: urn:request:5c42f53c-6a7a-432d-97d0-c881a3ee5e97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:14,238 [239] [DEBUG] [app] Starting request: urn:request:262acf17-d444-42a7-9b3c-426bfbd65e65 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:14,238 [239] [DEBUG] [app] Ending request: urn:request:262acf17-d444-42a7-9b3c-426bfbd65e65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:262acf17-d444-42a7-9b3c-426bfbd65e65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:14,238 [244] [DEBUG] [app] Ending request: urn:request:5c42f53c-6a7a-432d-97d0-c881a3ee5e97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c42f53c-6a7a-432d-97d0-c881a3ee5e97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:26:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:26:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:14,241 [235] [DEBUG] [app] Starting request: urn:request:ee779523-6b0f-4d9e-ba6e-800d28a9ae83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:14,242 [235] [DEBUG] [app] Ending request: urn:request:ee779523-6b0f-4d9e-ba6e-800d28a9ae83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee779523-6b0f-4d9e-ba6e-800d28a9ae83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:26:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:26:14,242 [238] [DEBUG] [app] Starting request: urn:request:53eefaab-7ada-4994-886c-bed03b472aaf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:14,242 [238] [DEBUG] [app] Ending request: urn:request:53eefaab-7ada-4994-886c-bed03b472aaf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53eefaab-7ada-4994-886c-bed03b472aaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:26:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:26:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:26:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:26:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:14,254 [238] [DEBUG] [app] Ending request: urn:request:a7021f0d-a504-4ac6-8202-78081cb55710 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7021f0d-a504-4ac6-8202-78081cb55710', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:26:14,254 [236] [DEBUG] [app] Ending request: urn:request:64b9a6e9-9de1-49d0-aa4c-ea6cba212bc1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:64b9a6e9-9de1-49d0-aa4c-ea6cba212bc1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:26:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:26:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:26:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:26:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:26:15,323 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:26:15,704 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:26:15,707 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:26:15,710 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:26:15,713 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:26:15,717 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:26:15,748 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:26:16,010 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:26:16,730 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:26:16,733 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:26:16,737 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:26:17,662 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:26:17,665 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:26:17,669 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:26:17,672 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:26:17,676 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:26:17,679 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:26:17,682 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:26:17,685 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:26:17,687 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:26:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:26:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:28.732147+00:00 (in 9.999421 seconds)
notificationworker stdout | 2025-02-07 03:26:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:28 UTC)" (scheduled at 2025-02-07 03:26:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:26:18,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:26:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 18, 733050), True, datetime.datetime(2025, 2, 7, 3, 26, 18, 733050), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:26:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:26:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:26:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:26:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:26:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:03.419110+00:00 (in 39.997746 seconds)
repositorygcworker stdout | 2025-02-07 03:26:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:23 UTC)" (scheduled at 2025-02-07 03:26:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:26:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:26:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:26:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:26:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:40.424250+00:00 (in 16.000733 seconds)
gcworker stdout | 2025-02-07 03:26:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:54 UTC)" (scheduled at 2025-02-07 03:26:24.423055+00:00)
gcworker stdout | 2025-02-07 03:26:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:26:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:26:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:26:24,984 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:26:26,939 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:26:27,935 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:26:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:26:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:38.732147+00:00 (in 9.999522 seconds)
notificationworker stdout | 2025-02-07 03:26:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:38 UTC)" (scheduled at 2025-02-07 03:26:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:26:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:26:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 28, 732905), True, datetime.datetime(2025, 2, 7, 3, 26, 28, 732905), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:26:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:26:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:26:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:26:29,236 [235] [DEBUG] [app] Starting request: urn:request:12287b4e-135e-4ef0-84fb-def63a47f94b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:29,236 [238] [DEBUG] [app] Starting request: urn:request:c4794549-8a25-4915-896d-76010722d2e3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:26:29,238 [239] [DEBUG] [app] Starting request: urn:request:01dbb06a-5d61-4986-beda-f7191116e04b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:29,238 [245] [DEBUG] [app] Starting request: urn:request:7dceba43-e3c8-4b74-99e6-8855113d66db (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:29,239 [239] [DEBUG] [app] Ending request: urn:request:01dbb06a-5d61-4986-beda-f7191116e04b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01dbb06a-5d61-4986-beda-f7191116e04b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:29,239 [245] [DEBUG] [app] Ending request: urn:request:7dceba43-e3c8-4b74-99e6-8855113d66db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7dceba43-e3c8-4b74-99e6-8855113d66db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:26:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:26:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:29,242 [235] [DEBUG] [app] Starting request: urn:request:e785bf3f-fb9e-44cb-b3ab-cb7919a628d8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:29,242 [236] [DEBUG] [app] Starting request: urn:request:57325f51-19ec-476a-8df9-3802e54da853 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:29,242 [235] [DEBUG] [app] Ending request: urn:request:e785bf3f-fb9e-44cb-b3ab-cb7919a628d8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e785bf3f-fb9e-44cb-b3ab-cb7919a628d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:26:29,242 [236] [DEBUG] [app] Ending request: urn:request:57325f51-19ec-476a-8df9-3802e54da853 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57325f51-19ec-476a-8df9-3802e54da853', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:26:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:26:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:26:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:26:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:26:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:26:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:29,254 [235] [DEBUG] [app] Ending request: urn:request:12287b4e-135e-4ef0-84fb-def63a47f94b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:12287b4e-135e-4ef0-84fb-def63a47f94b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:26:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:26:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:26:29,254 [238] [DEBUG] [app] Ending request: urn:request:c4794549-8a25-4915-896d-76010722d2e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4794549-8a25-4915-896d-76010722d2e3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:26:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:26:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:26:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:26:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:32.625945+00:00 (in 1.004286 seconds)
namespacegcworker stdout | 2025-02-07 03:26:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:31 UTC)" (scheduled at 2025-02-07 03:26:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:26:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:26:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 31, 621979), True, datetime.datetime(2025, 2, 7, 3, 26, 31, 621979), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:26:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:26:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:26:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:26:32,210 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:26:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:26:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:13.624301+00:00 (in 40.997815 seconds)
namespacegcworker stdout | 2025-02-07 03:26:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:32 UTC)" (scheduled at 2025-02-07 03:26:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:26:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:26:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:26:33,243 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:26:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:26:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:03.919286+00:00 (in 29.999491 seconds)
buildlogsarchiver stdout | 2025-02-07 03:26:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:03 UTC)" (scheduled at 2025-02-07 03:26:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:26:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 26, 33, 920057), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:26:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:26:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:26:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:56.713547+00:00 (in 18.997206 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:37 UTC)" (scheduled at 2025-02-07 03:26:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:26:37,917 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:26:38,387 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:26:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:26:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:48.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 03:26:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:48 UTC)" (scheduled at 2025-02-07 03:26:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:26:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:26:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 38, 732882), True, datetime.datetime(2025, 2, 7, 3, 26, 38, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:26:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:26:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:26:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:26:39,678 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:26:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:26:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:54.423055+00:00 (in 13.998332 seconds)
gcworker stdout | 2025-02-07 03:26:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:10 UTC)" (scheduled at 2025-02-07 03:26:40.424250+00:00)
gcworker stdout | 2025-02-07 03:26:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:26:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898500431, None, 1, 0])
gcworker stdout | 2025-02-07 03:26:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:26:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:26:40,448 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:26:40,524 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:26:41,659 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:26:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:26:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:11.831521+00:00 (in 29.999547 seconds)
autopruneworker stdout | 2025-02-07 03:26:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:11 UTC)" (scheduled at 2025-02-07 03:26:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:26:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895201836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:26:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:26:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:26:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:11 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:41,913 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:26:42,677 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:26:42,736 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:26:42,780 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:26:43,151 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:26:43,594 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:26:43,718 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:26:43,768 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:26:44,236 [235] [DEBUG] [app] Starting request: urn:request:1240fd62-1cd3-45ac-b6e7-9efae335c312 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:44,236 [237] [DEBUG] [app] Starting request: urn:request:c87c2131-7a22-400c-8c07-873a946a8780 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:44,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:26:44,239 [246] [DEBUG] [app] Starting request: urn:request:a353ecc9-52d9-4477-9e3c-5f416ed49bdf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:44,240 [245] [DEBUG] [app] Starting request: urn:request:fd4c5bf8-d01d-444e-91fa-60e4dfd28dfd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:44,240 [246] [DEBUG] [app] Ending request: urn:request:a353ecc9-52d9-4477-9e3c-5f416ed49bdf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a353ecc9-52d9-4477-9e3c-5f416ed49bdf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:44,240 [245] [DEBUG] [app] Ending request: urn:request:fd4c5bf8-d01d-444e-91fa-60e4dfd28dfd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd4c5bf8-d01d-444e-91fa-60e4dfd28dfd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:26:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:26:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:26:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:44,243 [235] [DEBUG] [app] Starting request: urn:request:5a80efb8-6768-474a-beeb-df1d71f5c965 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:44,243 [235] [DEBUG] [app] Ending request: urn:request:5a80efb8-6768-474a-beeb-df1d71f5c965 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a80efb8-6768-474a-beeb-df1d71f5c965', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:26:44,243 [238] [DEBUG] [app] Starting request: urn:request:fc85d93b-a740-4b61-96ed-2253fedb5338 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:26:44,244 [238] [DEBUG] [app] Ending request: urn:request:fc85d93b-a740-4b61-96ed-2253fedb5338 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc85d93b-a740-4b61-96ed-2253fedb5338', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:26:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:26:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:26:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:26:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:44,255 [235] [DEBUG] [app] Ending request: urn:request:1240fd62-1cd3-45ac-b6e7-9efae335c312 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1240fd62-1cd3-45ac-b6e7-9efae335c312', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:26:44,255 [237] [DEBUG] [app] Ending request: urn:request:c87c2131-7a22-400c-8c07-873a946a8780 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c87c2131-7a22-400c-8c07-873a946a8780', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:26:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:26:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:26:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:26:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:26:45,360 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:26:45,713 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:26:45,716 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:26:45,718 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:26:45,722 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:26:45,724 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:26:45,774 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:26:46,033 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:26:46,740 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:26:46,743 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:26:46,747 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:26:47,673 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:26:47,676 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:26:47,678 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:26:47,682 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:26:47,684 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:26:47,688 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:26:47,691 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:26:47,694 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:26:47,696 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:26:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:26:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:50.814677+00:00 (in 2.082081 seconds)
notificationworker stdout | 2025-02-07 03:26:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:58 UTC)" (scheduled at 2025-02-07 03:26:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:26:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:26:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 48, 732882), True, datetime.datetime(2025, 2, 7, 3, 26, 48, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:26:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:26:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:26:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:26:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:26:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:26:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:26:58.732147+00:00 (in 7.917018 seconds)
notificationworker stdout | 2025-02-07 03:26:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:50 UTC)" (scheduled at 2025-02-07 03:26:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:26:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:26:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:26:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:26:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 03:26:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:24 UTC)" (scheduled at 2025-02-07 03:26:54.423055+00:00)
gcworker stdout | 2025-02-07 03:26:54,423 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 03:26:54,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:26:54,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:26:54,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:26:55,018 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:26:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:26:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:56.118855+00:00 (in 59.999522 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:26:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:56 UTC)" (scheduled at 2025-02-07 03:26:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:26:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:26:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:26:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:37.715878+00:00 (in 41.001884 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:56 UTC)" (scheduled at 2025-02-07 03:26:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:26:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:26:56,946 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:26:57,971 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:26:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:26:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:08.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 03:26:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:08 UTC)" (scheduled at 2025-02-07 03:26:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:26:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:26:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 26, 58, 732858), True, datetime.datetime(2025, 2, 7, 3, 26, 58, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:26:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:26:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:26:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:26:59,235 [238] [DEBUG] [app] Starting request: urn:request:e101d95f-5e5d-4651-a41f-95615cd0555e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:59,235 [236] [DEBUG] [app] Starting request: urn:request:bf37033e-2f57-458c-922a-ca385f5a1a9f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:26:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:26:59,239 [246] [DEBUG] [app] Starting request: urn:request:58f6ef13-3b13-4518-8917-c35622d42c06 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:59,239 [239] [DEBUG] [app] Starting request: urn:request:8fcdf09e-d124-4b84-a3a1-16f33c0eb28a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:26:59,239 [239] [DEBUG] [app] Ending request: urn:request:8fcdf09e-d124-4b84-a3a1-16f33c0eb28a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fcdf09e-d124-4b84-a3a1-16f33c0eb28a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:59,239 [246] [DEBUG] [app] Ending request: urn:request:58f6ef13-3b13-4518-8917-c35622d42c06 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58f6ef13-3b13-4518-8917-c35622d42c06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:26:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:26:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 03:26:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:26:59,242 [238] [DEBUG] [app] Starting request: urn:request:4b94b76f-899c-4a2a-876c-e67eae0d8a66 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:59,243 [238] [DEBUG] [app] Ending request: urn:request:4b94b76f-899c-4a2a-876c-e67eae0d8a66 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b94b76f-899c-4a2a-876c-e67eae0d8a66', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:26:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:26:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:26:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:26:59,244 [237] [DEBUG] [app] Starting request: urn:request:6b7502d1-fc38-4c95-a26e-50a5eb60f9e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:26:59,245 [237] [DEBUG] [app] Ending request: urn:request:6b7502d1-fc38-4c95-a26e-50a5eb60f9e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b7502d1-fc38-4c95-a26e-50a5eb60f9e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:26:59,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:26:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:26:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:26:59,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:26:59,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:26:59,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:26:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:59,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:26:59,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:26:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:59,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:26:59,254 [236] [DEBUG] [app] Ending request: urn:request:bf37033e-2f57-458c-922a-ca385f5a1a9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf37033e-2f57-458c-922a-ca385f5a1a9f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:26:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:26:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:26:59,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:26:59,256 [238] [DEBUG] [app] Ending request: urn:request:e101d95f-5e5d-4651-a41f-95615cd0555e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e101d95f-5e5d-4651-a41f-95615cd0555e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:26:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:26:59,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:26:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:26:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
exportactionlogsworker stdout | 2025-02-07 03:26:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:26:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:04.936688+00:00 (in 4.998300 seconds)
exportactionlogsworker stdout | 2025-02-07 03:26:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:59 UTC)" (scheduled at 2025-02-07 03:26:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:26:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:26:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:27:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:27:02,247 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:27:03,263 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:27:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:27:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:23.420886+00:00 (in 20.001331 seconds)
repositorygcworker stdout | 2025-02-07 03:27:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:03 UTC)" (scheduled at 2025-02-07 03:27:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:27:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:27:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 3, 419831), True, datetime.datetime(2025, 2, 7, 3, 27, 3, 419831), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:27:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:27:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:27:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:27:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:27:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:33.919286+00:00 (in 29.999521 seconds)
buildlogsarchiver stdout | 2025-02-07 03:27:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:33 UTC)" (scheduled at 2025-02-07 03:27:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:27:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 27, 3, 920027), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:27:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:27:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:27:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:27:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:27:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:59.937899+00:00 (in 55.000781 seconds)
exportactionlogsworker stdout | 2025-02-07 03:27:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:04 UTC)" (scheduled at 2025-02-07 03:27:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:27:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:27:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 4, 937427), True, datetime.datetime(2025, 2, 7, 3, 27, 4, 937427), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:27:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:27:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:27:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:27:07,925 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:27:08,424 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:27:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:18.732147+00:00 (in 9.999446 seconds)
notificationworker stdout | 2025-02-07 03:27:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:18 UTC)" (scheduled at 2025-02-07 03:27:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:27:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:27:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 8, 733081), True, datetime.datetime(2025, 2, 7, 3, 27, 8, 733081), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:27:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:27:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:27:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:27:09,711 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:27:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:27:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:24.423055+00:00 (in 13.998357 seconds)
gcworker stdout | 2025-02-07 03:27:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:40 UTC)" (scheduled at 2025-02-07 03:27:10.424250+00:00)
gcworker stdout | 2025-02-07 03:27:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:27:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898530430, None, 1, 0])
gcworker stdout | 2025-02-07 03:27:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:27:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:27:10,486 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:27:10,560 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:27:11,695 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:27:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:27:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 03:27:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:41 UTC)" (scheduled at 2025-02-07 03:27:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:27:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895231836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:27:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:27:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:27:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:41 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:11,949 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:27:12,706 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:27:12,770 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:27:12,794 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:27:13,187 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
namespacegcworker stdout | 2025-02-07 03:27:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:27:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:31.621131+00:00 (in 17.996341 seconds)
namespacegcworker stdout | 2025-02-07 03:27:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:32:13 UTC)" (scheduled at 2025-02-07 03:27:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:27:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 27, 13, 625050), 'namespacegc/%'])
manifestbackfillworker stdout | 2025-02-07 03:27:13,631 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 03:27:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 27, 13, 625050), True, datetime.datetime(2025, 2, 7, 3, 27, 13, 625050), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:27:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 27, 13, 625050), True, datetime.datetime(2025, 2, 7, 3, 27, 13, 625050), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 27, 13, 625050), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:27:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:27:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:32:13 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:27:13,756 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:27:13,796 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:27:14,236 [238] [DEBUG] [app] Starting request: urn:request:6e7644ca-4567-4028-984c-d4f645b3e34e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:14,236 [237] [DEBUG] [app] Starting request: urn:request:1a8c1a99-08c5-4176-b1fe-9b0186db73ac (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:14,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:14,239 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:27:14,240 [246] [DEBUG] [app] Starting request: urn:request:45f16933-317a-4e06-bffb-9624c999ff17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:27:14,241 [246] [DEBUG] [app] Ending request: urn:request:45f16933-317a-4e06-bffb-9624c999ff17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:45f16933-317a-4e06-bffb-9624c999ff17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:27:14,241 [239] [DEBUG] [app] Starting request: urn:request:c08e43ea-126e-42b4-8952-b3d8c4f1d72f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:27:14,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:27:14,242 [239] [DEBUG] [app] Ending request: urn:request:c08e43ea-126e-42b4-8952-b3d8c4f1d72f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c08e43ea-126e-42b4-8952-b3d8c4f1d72f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:27:14,242 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:14,243 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:14,245 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:14,245 [235] [DEBUG] [app] Starting request: urn:request:108825dc-a7cf-4454-aa2d-be84a7cda1cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:14,245 [235] [DEBUG] [app] Ending request: urn:request:108825dc-a7cf-4454-aa2d-be84a7cda1cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:108825dc-a7cf-4454-aa2d-be84a7cda1cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:27:14,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:14,246 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:27:14,246 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:14,246 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:14,246 [236] [DEBUG] [app] Starting request: urn:request:15bfd091-e9dc-45e9-862d-1a512a45f5c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:14,247 [236] [DEBUG] [app] Ending request: urn:request:15bfd091-e9dc-45e9-862d-1a512a45f5c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:15bfd091-e9dc-45e9-862d-1a512a45f5c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:27:14,247 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:14,247 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:14,248 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:14,248 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:14,250 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:14,250 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:14,251 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:14,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:14,255 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:14,255 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:14,256 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:14,257 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:14,258 [237] [DEBUG] [app] Ending request: urn:request:1a8c1a99-08c5-4176-b1fe-9b0186db73ac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1a8c1a99-08c5-4176-b1fe-9b0186db73ac', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:14,258 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:27:14,258 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.022)
gunicorn-web stdout | 2025-02-07 03:27:14,259 [238] [DEBUG] [app] Ending request: urn:request:6e7644ca-4567-4028-984c-d4f645b3e34e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e7644ca-4567-4028-984c-d4f645b3e34e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:14,259 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:27:14,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
namespacegcworker stdout | 2025-02-07 03:27:15,393 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:27:15,721 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:27:15,724 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:27:15,727 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:27:15,729 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:27:15,731 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:27:15,810 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:27:16,063 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:27:16,750 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:27:16,752 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:27:16,755 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:27:17,684 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:27:17,687 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:27:17,690 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:27:17,693 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:27:17,696 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:27:17,699 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:27:17,702 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:27:17,705 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:27:17,707 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:27:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:28.732147+00:00 (in 9.999512 seconds)
notificationworker stdout | 2025-02-07 03:27:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:28 UTC)" (scheduled at 2025-02-07 03:27:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:27:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:27:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 18, 732920), True, datetime.datetime(2025, 2, 7, 3, 27, 18, 732920), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:27:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:27:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:27:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:27:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:27:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:03.419110+00:00 (in 39.997680 seconds)
repositorygcworker stdout | 2025-02-07 03:27:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:23 UTC)" (scheduled at 2025-02-07 03:27:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:27:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:27:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:27:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:27:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:40.424250+00:00 (in 16.000720 seconds)
gcworker stdout | 2025-02-07 03:27:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:54 UTC)" (scheduled at 2025-02-07 03:27:24.423055+00:00)
gcworker stdout | 2025-02-07 03:27:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:27:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:27:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:27:25,036 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:27:26,970 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:27:28,001 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:27:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:33.814166+00:00 (in 5.081567 seconds)
notificationworker stdout | 2025-02-07 03:27:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:38 UTC)" (scheduled at 2025-02-07 03:27:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:27:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:27:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 28, 732799), True, datetime.datetime(2025, 2, 7, 3, 27, 28, 732799), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:27:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:27:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:27:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:27:29,236 [236] [DEBUG] [app] Starting request: urn:request:b536e0c7-243c-4093-860d-a29d43c17aa8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:29,236 [238] [DEBUG] [app] Starting request: urn:request:37cc552e-d4f5-4367-9a04-4579d96ce7a1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:27:29,239 [246] [DEBUG] [app] Starting request: urn:request:1b311e3c-a7cb-47bf-af2c-e0e288e5aebf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:27:29,239 [239] [DEBUG] [app] Starting request: urn:request:79248313-2865-4ad7-a5fd-647f405e87cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:27:29,239 [239] [DEBUG] [app] Ending request: urn:request:79248313-2865-4ad7-a5fd-647f405e87cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:79248313-2865-4ad7-a5fd-647f405e87cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:27:29,239 [246] [DEBUG] [app] Ending request: urn:request:1b311e3c-a7cb-47bf-af2c-e0e288e5aebf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b311e3c-a7cb-47bf-af2c-e0e288e5aebf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:27:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:27:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:27:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:29,242 [235] [DEBUG] [app] Starting request: urn:request:4327fac5-f715-4b78-aa6f-90cc4c28df32 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:29,242 [237] [DEBUG] [app] Starting request: urn:request:c6ae4062-591f-4785-b621-5418c657b429 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:29,242 [235] [DEBUG] [app] Ending request: urn:request:4327fac5-f715-4b78-aa6f-90cc4c28df32 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4327fac5-f715-4b78-aa6f-90cc4c28df32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:27:29,242 [237] [DEBUG] [app] Ending request: urn:request:c6ae4062-591f-4785-b621-5418c657b429 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6ae4062-591f-4785-b621-5418c657b429', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:27:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:29,255 [236] [DEBUG] [app] Ending request: urn:request:b536e0c7-243c-4093-860d-a29d43c17aa8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b536e0c7-243c-4093-860d-a29d43c17aa8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:29,255 [238] [DEBUG] [app] Ending request: urn:request:37cc552e-d4f5-4367-9a04-4579d96ce7a1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37cc552e-d4f5-4367-9a04-4579d96ce7a1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:27:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:27:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:27:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:27:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:27:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:32.625945+00:00 (in 1.004314 seconds)
namespacegcworker stdout | 2025-02-07 03:27:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:31 UTC)" (scheduled at 2025-02-07 03:27:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:27:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:27:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 31, 621949), True, datetime.datetime(2025, 2, 7, 3, 27, 31, 621949), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:27:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:27:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:27:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:27:32,283 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:27:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:27:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:31.621131+00:00 (in 58.994624 seconds)
namespacegcworker stdout | 2025-02-07 03:27:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:32 UTC)" (scheduled at 2025-02-07 03:27:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:27:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:27:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:27:33,300 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 03:27:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:38.732147+00:00 (in 4.917535 seconds)
notificationworker stdout | 2025-02-07 03:27:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:32:33 UTC)" (scheduled at 2025-02-07 03:27:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:27:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 27, 33, 814793), 'notification/%'])
notificationworker stdout | 2025-02-07 03:27:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 27, 33, 814793), True, datetime.datetime(2025, 2, 7, 3, 27, 33, 814793), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:27:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 27, 33, 814793), True, datetime.datetime(2025, 2, 7, 3, 27, 33, 814793), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 27, 33, 814793), 'notification/%'])
notificationworker stdout | 2025-02-07 03:27:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:27:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:32:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:27:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:27:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:03.919286+00:00 (in 29.999574 seconds)
buildlogsarchiver stdout | 2025-02-07 03:27:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:03 UTC)" (scheduled at 2025-02-07 03:27:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:27:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 27, 33, 919951), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:27:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:27:33,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:27:33,926 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:56.713547+00:00 (in 18.997189 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:37 UTC)" (scheduled at 2025-02-07 03:27:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:27:37,942 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:27:38,458 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:27:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:48.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 03:27:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:48 UTC)" (scheduled at 2025-02-07 03:27:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:27:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:27:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 38, 732815), True, datetime.datetime(2025, 2, 7, 3, 27, 38, 732815), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:27:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:27:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:27:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:27:39,734 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:27:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:27:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:54.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 03:27:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:10 UTC)" (scheduled at 2025-02-07 03:27:40.424250+00:00)
gcworker stdout | 2025-02-07 03:27:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:27:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898560431, None, 1, 0])
gcworker stdout | 2025-02-07 03:27:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:27:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:27:40,517 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:27:40,596 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:27:41,719 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:27:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:27:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:11.831521+00:00 (in 29.999553 seconds)
autopruneworker stdout | 2025-02-07 03:27:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:11 UTC)" (scheduled at 2025-02-07 03:27:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:27:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895261836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:27:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:27:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:27:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:11 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:41,978 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:27:42,723 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:27:42,795 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:27:42,830 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:27:43,223 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:27:43,667 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:27:43,795 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:27:43,825 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:27:44,236 [236] [DEBUG] [app] Starting request: urn:request:df3aaef3-314e-4bc5-8881-c1bae801bde4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:44,236 [235] [DEBUG] [app] Starting request: urn:request:9839a526-94fd-46ce-a469-231c0cf3a3e2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:27:44,239 [246] [DEBUG] [app] Starting request: urn:request:050a11a8-d3d4-453d-aaf1-6d78b4f8147d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:27:44,239 [244] [DEBUG] [app] Starting request: urn:request:adf09397-f5b5-4251-82da-99a1f1f70070 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:27:44,239 [244] [DEBUG] [app] Ending request: urn:request:adf09397-f5b5-4251-82da-99a1f1f70070 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:adf09397-f5b5-4251-82da-99a1f1f70070', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:27:44,239 [246] [DEBUG] [app] Ending request: urn:request:050a11a8-d3d4-453d-aaf1-6d78b4f8147d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:050a11a8-d3d4-453d-aaf1-6d78b4f8147d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:27:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:27:44,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:27:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:44,242 [235] [DEBUG] [app] Starting request: urn:request:f0d2dceb-ae55-445c-a235-a5cf85a3c95e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:44,242 [236] [DEBUG] [app] Starting request: urn:request:6ae29544-56cf-4ff9-97f6-98cea4de5608 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:44,243 [235] [DEBUG] [app] Ending request: urn:request:f0d2dceb-ae55-445c-a235-a5cf85a3c95e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f0d2dceb-ae55-445c-a235-a5cf85a3c95e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:27:44,243 [236] [DEBUG] [app] Ending request: urn:request:6ae29544-56cf-4ff9-97f6-98cea4de5608 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6ae29544-56cf-4ff9-97f6-98cea4de5608', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:27:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:27:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:27:44,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:44,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:44,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:44,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:44,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:44,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:44,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:44,255 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:44,256 [236] [DEBUG] [app] Ending request: urn:request:df3aaef3-314e-4bc5-8881-c1bae801bde4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df3aaef3-314e-4bc5-8881-c1bae801bde4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:44,256 [235] [DEBUG] [app] Ending request: urn:request:9839a526-94fd-46ce-a469-231c0cf3a3e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9839a526-94fd-46ce-a469-231c0cf3a3e2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:44,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:27:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:27:44,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:27:44,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
namespacegcworker stdout | 2025-02-07 03:27:45,429 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:27:45,729 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:27:45,732 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:27:45,735 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:27:45,738 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:27:45,741 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:27:45,842 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:27:46,100 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:27:46,759 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:27:46,762 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:27:46,764 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:27:47,692 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:27:47,694 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:27:47,699 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:27:47,702 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:27:47,704 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:27:47,708 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:27:47,711 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:27:47,714 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:27:47,717 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:27:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:50.814677+00:00 (in 2.082050 seconds)
notificationworker stdout | 2025-02-07 03:27:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:58 UTC)" (scheduled at 2025-02-07 03:27:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:27:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:27:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 48, 732847), True, datetime.datetime(2025, 2, 7, 3, 27, 48, 732847), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:27:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:27:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:27:48,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:27:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:27:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:27:58.732147+00:00 (in 7.917009 seconds)
notificationworker stdout | 2025-02-07 03:27:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:50 UTC)" (scheduled at 2025-02-07 03:27:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:27:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:27:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:27:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:27:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:10.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 03:27:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:24 UTC)" (scheduled at 2025-02-07 03:27:54.423055+00:00)
gcworker stdout | 2025-02-07 03:27:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:27:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:27:55,050 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:27:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:27:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:56.118855+00:00 (in 59.999518 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:27:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:56 UTC)" (scheduled at 2025-02-07 03:27:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:27:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:27:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:27:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:37.715878+00:00 (in 41.001891 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:56 UTC)" (scheduled at 2025-02-07 03:27:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:27:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:27:57,006 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:27:58,016 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:27:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:27:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:08.732147+00:00 (in 9.999528 seconds)
notificationworker stdout | 2025-02-07 03:27:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:08 UTC)" (scheduled at 2025-02-07 03:27:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:27:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:27:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 27, 58, 732916), True, datetime.datetime(2025, 2, 7, 3, 27, 58, 732916), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:27:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:27:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:27:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:27:59,235 [237] [DEBUG] [app] Starting request: urn:request:f9076703-2373-428f-ab3d-670104af1f6d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:59,235 [236] [DEBUG] [app] Starting request: urn:request:60af6beb-0493-4b26-b9dd-f47ac5f41ce8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:27:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:27:59,238 [239] [DEBUG] [app] Starting request: urn:request:32c6ea74-f575-42f5-bd9e-f1988368140f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:27:59,238 [246] [DEBUG] [app] Starting request: urn:request:3a9ade97-8ed0-4a8f-931f-3f0758137792 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:27:59,239 [239] [DEBUG] [app] Ending request: urn:request:32c6ea74-f575-42f5-bd9e-f1988368140f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:32c6ea74-f575-42f5-bd9e-f1988368140f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:27:59,239 [246] [DEBUG] [app] Ending request: urn:request:3a9ade97-8ed0-4a8f-931f-3f0758137792 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a9ade97-8ed0-4a8f-931f-3f0758137792', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:27:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:27:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:27:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:27:59,241 [238] [DEBUG] [app] Starting request: urn:request:ebf7ebd6-d51b-4849-963d-6df2e51cc1b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:59,242 [238] [DEBUG] [app] Ending request: urn:request:ebf7ebd6-d51b-4849-963d-6df2e51cc1b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ebf7ebd6-d51b-4849-963d-6df2e51cc1b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:27:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:27:59,243 [237] [DEBUG] [app] Starting request: urn:request:92fc1345-977f-4c65-b25a-1899579c3ac7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:27:59,243 [237] [DEBUG] [app] Ending request: urn:request:92fc1345-977f-4c65-b25a-1899579c3ac7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92fc1345-977f-4c65-b25a-1899579c3ac7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:27:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:27:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:27:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:27:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:27:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:27:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:27:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:59,263 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:27:59,263 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:27:59,265 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:59,266 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:59,267 [237] [DEBUG] [app] Ending request: urn:request:f9076703-2373-428f-ab3d-670104af1f6d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f9076703-2373-428f-ab3d-670104af1f6d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:59,267 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:27:59,267 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:27:59,268 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.034 118 0.034)
gunicorn-web stdout | 2025-02-07 03:27:59,269 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:27:59,270 [236] [DEBUG] [app] Ending request: urn:request:60af6beb-0493-4b26-b9dd-f47ac5f41ce8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:60af6beb-0493-4b26-b9dd-f47ac5f41ce8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:27:59,270 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:27:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.036 118 0.036)
gunicorn-web stdout | 2025-02-07 03:27:59,271 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:27:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:27:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:27:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:04.936688+00:00 (in 4.998330 seconds)
exportactionlogsworker stdout | 2025-02-07 03:27:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:59 UTC)" (scheduled at 2025-02-07 03:27:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:27:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:27:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:28:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:28:02,314 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:28:03,334 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:28:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:28:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:23.420886+00:00 (in 20.001276 seconds)
repositorygcworker stdout | 2025-02-07 03:28:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:03 UTC)" (scheduled at 2025-02-07 03:28:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:28:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:28:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 3, 419880), True, datetime.datetime(2025, 2, 7, 3, 28, 3, 419880), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:28:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:28:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:28:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:28:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:28:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:33.919286+00:00 (in 29.999551 seconds)
buildlogsarchiver stdout | 2025-02-07 03:28:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:33 UTC)" (scheduled at 2025-02-07 03:28:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:28:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 28, 3, 919988), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:28:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:28:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:28:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:28:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:28:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:59.937899+00:00 (in 55.000747 seconds)
exportactionlogsworker stdout | 2025-02-07 03:28:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:04 UTC)" (scheduled at 2025-02-07 03:28:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:28:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:28:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 4, 937487), True, datetime.datetime(2025, 2, 7, 3, 28, 4, 937487), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:28:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:28:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:28:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:28:07,968 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:28:08,495 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:28:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:28:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:18.732147+00:00 (in 9.999391 seconds)
notificationworker stdout | 2025-02-07 03:28:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:18 UTC)" (scheduled at 2025-02-07 03:28:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:28:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:28:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 8, 733150), True, datetime.datetime(2025, 2, 7, 3, 28, 8, 733150), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:28:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:28:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:28:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:28:09,752 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:28:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:28:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:24.423055+00:00 (in 13.998350 seconds)
gcworker stdout | 2025-02-07 03:28:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:40 UTC)" (scheduled at 2025-02-07 03:28:10.424250+00:00)
gcworker stdout | 2025-02-07 03:28:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:28:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898590431, None, 1, 0])
gcworker stdout | 2025-02-07 03:28:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:28:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:28:10,553 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:28:10,632 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:28:11,736 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:28:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:28:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:41.831521+00:00 (in 29.999562 seconds)
autopruneworker stdout | 2025-02-07 03:28:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:41 UTC)" (scheduled at 2025-02-07 03:28:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:28:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895291836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:28:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:28:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:28:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:41 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:12,014 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:28:12,741 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:28:12,817 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:28:12,866 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:28:13,235 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:28:13,703 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:28:13,803 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:28:13,861 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:28:14,236 [236] [DEBUG] [app] Starting request: urn:request:01366682-74df-445c-9023-15868cf39807 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:14,236 [238] [DEBUG] [app] Starting request: urn:request:048933e9-fe88-4e3f-881b-18bab8ebc22d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:28:14,239 [239] [DEBUG] [app] Starting request: urn:request:a3728a53-e93d-4e1c-8545-02973d5f74a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:14,239 [245] [DEBUG] [app] Starting request: urn:request:ad1c784a-8999-4c37-962d-380dbbaf1ffd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:14,239 [245] [DEBUG] [app] Ending request: urn:request:ad1c784a-8999-4c37-962d-380dbbaf1ffd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ad1c784a-8999-4c37-962d-380dbbaf1ffd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:14,239 [239] [DEBUG] [app] Ending request: urn:request:a3728a53-e93d-4e1c-8545-02973d5f74a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3728a53-e93d-4e1c-8545-02973d5f74a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:28:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 03:28:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:14,242 [235] [DEBUG] [app] Starting request: urn:request:05c5a45f-c0a4-432d-b4a3-e6f83c585e24 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:14,242 [238] [DEBUG] [app] Starting request: urn:request:df5a44d2-f5cd-4b44-aa97-eead311210a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:14,243 [235] [DEBUG] [app] Ending request: urn:request:05c5a45f-c0a4-432d-b4a3-e6f83c585e24 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05c5a45f-c0a4-432d-b4a3-e6f83c585e24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:28:14,243 [238] [DEBUG] [app] Ending request: urn:request:df5a44d2-f5cd-4b44-aa97-eead311210a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df5a44d2-f5cd-4b44-aa97-eead311210a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:28:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:14,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:14,254 [238] [DEBUG] [app] Ending request: urn:request:048933e9-fe88-4e3f-881b-18bab8ebc22d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:048933e9-fe88-4e3f-881b-18bab8ebc22d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:14,255 [236] [DEBUG] [app] Ending request: urn:request:01366682-74df-445c-9023-15868cf39807 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01366682-74df-445c-9023-15868cf39807', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:28:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:28:15,465 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:28:15,738 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:28:15,741 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:28:15,744 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:28:15,746 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:28:15,749 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:28:15,878 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:28:16,135 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:28:16,767 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:28:16,770 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:28:16,773 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:28:17,701 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:28:17,704 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:28:17,707 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:28:17,710 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:28:17,714 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:28:17,717 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:28:17,720 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:28:17,724 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:28:17,726 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 03:28:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:28:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:28.732147+00:00 (in 9.999508 seconds)
notificationworker stdout | 2025-02-07 03:28:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:28 UTC)" (scheduled at 2025-02-07 03:28:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:28:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:28:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 18, 732950), True, datetime.datetime(2025, 2, 7, 3, 28, 18, 732950), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:28:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:28:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:28:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:28:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:28:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:32.420707+00:00 (in 8.999313 seconds)
repositorygcworker stdout | 2025-02-07 03:28:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:23 UTC)" (scheduled at 2025-02-07 03:28:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:28:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:28:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:28:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:28:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:40.424250+00:00 (in 16.000650 seconds)
gcworker stdout | 2025-02-07 03:28:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:54 UTC)" (scheduled at 2025-02-07 03:28:24.423055+00:00)
gcworker stdout | 2025-02-07 03:28:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:28:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:28:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:28:25,078 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:28:27,038 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:28:28,052 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:28:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:28:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:38.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 03:28:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:38 UTC)" (scheduled at 2025-02-07 03:28:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:28:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:28:28,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 28, 732930), True, datetime.datetime(2025, 2, 7, 3, 28, 28, 732930), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:28:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:28:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:28:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:28:29,235 [236] [DEBUG] [app] Starting request: urn:request:8f584164-c6ec-4b4d-afbe-841d6740760d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:29,235 [237] [DEBUG] [app] Starting request: urn:request:d4e06785-91b6-4a7a-9408-c4373470c769 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:28:29,238 [246] [DEBUG] [app] Starting request: urn:request:4de301cd-3317-4e13-972c-f88fb83c5328 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:29,238 [243] [DEBUG] [app] Starting request: urn:request:efc221d6-3c34-4b0c-a1ed-67ba658ca5d8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:29,239 [246] [DEBUG] [app] Ending request: urn:request:4de301cd-3317-4e13-972c-f88fb83c5328 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4de301cd-3317-4e13-972c-f88fb83c5328', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:29,239 [243] [DEBUG] [app] Ending request: urn:request:efc221d6-3c34-4b0c-a1ed-67ba658ca5d8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:efc221d6-3c34-4b0c-a1ed-67ba658ca5d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:28:29,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:29,241 [237] [DEBUG] [app] Starting request: urn:request:f9d28e10-b721-4c66-862b-6c0c8cf9367c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:29,242 [237] [DEBUG] [app] Ending request: urn:request:f9d28e10-b721-4c66-862b-6c0c8cf9367c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9d28e10-b721-4c66-862b-6c0c8cf9367c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:28:29,242 [235] [DEBUG] [app] Starting request: urn:request:70f2ec15-1a52-4adb-830d-a8fa9ee6914c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:28:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:29,242 [235] [DEBUG] [app] Ending request: urn:request:70f2ec15-1a52-4adb-830d-a8fa9ee6914c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70f2ec15-1a52-4adb-830d-a8fa9ee6914c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:28:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:28:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:29,254 [236] [DEBUG] [app] Ending request: urn:request:8f584164-c6ec-4b4d-afbe-841d6740760d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f584164-c6ec-4b4d-afbe-841d6740760d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:29,254 [237] [DEBUG] [app] Ending request: urn:request:d4e06785-91b6-4a7a-9408-c4373470c769 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d4e06785-91b6-4a7a-9408-c4373470c769', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:28:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
namespacegcworker stdout | 2025-02-07 03:28:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:28:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:32.625945+00:00 (in 1.004326 seconds)
namespacegcworker stdout | 2025-02-07 03:28:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:31 UTC)" (scheduled at 2025-02-07 03:28:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:28:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:28:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 31, 621897), True, datetime.datetime(2025, 2, 7, 3, 28, 31, 621897), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:28:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:28:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:28:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:28:32,350 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:28:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:28:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:03.419110+00:00 (in 30.997972 seconds)
repositorygcworker stdout | 2025-02-07 03:28:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:33:32 UTC)" (scheduled at 2025-02-07 03:28:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:28:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 28, 32, 421406), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:28:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 28, 32, 421406), True, datetime.datetime(2025, 2, 7, 3, 28, 32, 421406), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:28:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 28, 32, 421406), True, datetime.datetime(2025, 2, 7, 3, 28, 32, 421406), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 28, 32, 421406), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:28:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:28:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:33:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:28:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:28:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:31.621131+00:00 (in 58.994729 seconds)
namespacegcworker stdout | 2025-02-07 03:28:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:32 UTC)" (scheduled at 2025-02-07 03:28:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:28:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:28:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:28:33,365 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:28:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:28:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:03.919286+00:00 (in 29.999530 seconds)
buildlogsarchiver stdout | 2025-02-07 03:28:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:03 UTC)" (scheduled at 2025-02-07 03:28:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:28:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 28, 33, 920010), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:28:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:28:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:28:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:56.713547+00:00 (in 18.997200 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:37 UTC)" (scheduled at 2025-02-07 03:28:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:28:38,005 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:28:38,523 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:28:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:28:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:48.732147+00:00 (in 9.999426 seconds)
notificationworker stdout | 2025-02-07 03:28:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:48 UTC)" (scheduled at 2025-02-07 03:28:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:28:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:28:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 38, 733093), True, datetime.datetime(2025, 2, 7, 3, 28, 38, 733093), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:28:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:28:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:28:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:28:39,782 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:28:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:28:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:54.423055+00:00 (in 13.998300 seconds)
gcworker stdout | 2025-02-07 03:28:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:10 UTC)" (scheduled at 2025-02-07 03:28:40.424250+00:00)
gcworker stdout | 2025-02-07 03:28:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:28:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898620431, None, 1, 0])
gcworker stdout | 2025-02-07 03:28:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:28:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:28:40,589 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:28:40,668 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:28:41,744 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:28:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:28:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:11.831521+00:00 (in 29.999533 seconds)
autopruneworker stdout | 2025-02-07 03:28:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:11 UTC)" (scheduled at 2025-02-07 03:28:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:28:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895321836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:28:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:28:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:28:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:11 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:42,051 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:28:42,775 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:28:42,853 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:28:42,903 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:28:43,272 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:28:43,740 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:28:43,831 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:28:43,882 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:28:44,235 [235] [DEBUG] [app] Starting request: urn:request:265d8762-12fd-4733-96fa-59a87a94f8a2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:44,235 [238] [DEBUG] [app] Starting request: urn:request:57f9796d-97d1-47ca-9336-571d35b8d37c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:28:44,238 [239] [DEBUG] [app] Starting request: urn:request:3b4a7aa1-fb3f-4821-b0ac-3f43649b9c85 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:44,238 [245] [DEBUG] [app] Starting request: urn:request:c0e06130-81b1-4a5a-a297-4943060e4bb2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:44,239 [245] [DEBUG] [app] Ending request: urn:request:c0e06130-81b1-4a5a-a297-4943060e4bb2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c0e06130-81b1-4a5a-a297-4943060e4bb2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:44,239 [239] [DEBUG] [app] Ending request: urn:request:3b4a7aa1-fb3f-4821-b0ac-3f43649b9c85 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b4a7aa1-fb3f-4821-b0ac-3f43649b9c85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:28:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:28:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:44,242 [237] [DEBUG] [app] Starting request: urn:request:4bfd095c-81ed-448e-978e-4c4c753105f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:44,242 [236] [DEBUG] [app] Starting request: urn:request:3d885b4b-9df6-4ae7-88be-a1a71d118801 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:44,242 [237] [DEBUG] [app] Ending request: urn:request:4bfd095c-81ed-448e-978e-4c4c753105f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4bfd095c-81ed-448e-978e-4c4c753105f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:28:44,242 [236] [DEBUG] [app] Ending request: urn:request:3d885b4b-9df6-4ae7-88be-a1a71d118801 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d885b4b-9df6-4ae7-88be-a1a71d118801', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:28:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:28:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:28:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:44,255 [238] [DEBUG] [app] Ending request: urn:request:57f9796d-97d1-47ca-9336-571d35b8d37c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:57f9796d-97d1-47ca-9336-571d35b8d37c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:44,255 [235] [DEBUG] [app] Ending request: urn:request:265d8762-12fd-4733-96fa-59a87a94f8a2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:265d8762-12fd-4733-96fa-59a87a94f8a2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:28:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:28:45,502 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:28:45,747 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:28:45,750 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:28:45,753 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:28:45,756 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:28:45,759 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:28:45,914 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:28:46,170 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:28:46,776 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:28:46,779 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:28:46,781 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:28:47,708 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:28:47,717 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:28:47,720 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:28:47,723 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:28:47,726 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:28:47,730 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:28:47,733 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:28:47,736 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:28:47,738 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:28:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:28:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:50.814677+00:00 (in 2.082054 seconds)
notificationworker stdout | 2025-02-07 03:28:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:58 UTC)" (scheduled at 2025-02-07 03:28:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:28:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:28:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 48, 732912), True, datetime.datetime(2025, 2, 7, 3, 28, 48, 732912), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:28:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:28:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:28:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:28:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:28:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:28:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:28:58.732147+00:00 (in 7.917014 seconds)
notificationworker stdout | 2025-02-07 03:28:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:50 UTC)" (scheduled at 2025-02-07 03:28:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:28:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:28:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:28:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:28:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:10.424250+00:00 (in 16.000708 seconds)
gcworker stdout | 2025-02-07 03:28:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:24 UTC)" (scheduled at 2025-02-07 03:28:54.423055+00:00)
gcworker stdout | 2025-02-07 03:28:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:28:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:28:55,088 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:28:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:28:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:56.118855+00:00 (in 59.999490 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:28:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:56 UTC)" (scheduled at 2025-02-07 03:28:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:28:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:28:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:28:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:37.715878+00:00 (in 41.001862 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:56 UTC)" (scheduled at 2025-02-07 03:28:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:28:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:28:57,074 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:28:58,076 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:28:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:28:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:08.732147+00:00 (in 9.999530 seconds)
notificationworker stdout | 2025-02-07 03:28:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:08 UTC)" (scheduled at 2025-02-07 03:28:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:28:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:28:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 28, 58, 732901), True, datetime.datetime(2025, 2, 7, 3, 28, 58, 732901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:28:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:28:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:28:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:28:59,235 [238] [DEBUG] [app] Starting request: urn:request:d991a080-7129-4444-bc0f-a4276eddae53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:59,235 [237] [DEBUG] [app] Starting request: urn:request:63cbd0fc-d507-4afb-aa54-c5e163712316 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:28:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:28:59,239 [239] [DEBUG] [app] Starting request: urn:request:2c8333e2-d55c-4a50-bd3b-f5511b94504d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:59,239 [246] [DEBUG] [app] Starting request: urn:request:ec333582-fa83-4149-86b8-6e094694cac5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:28:59,240 [239] [DEBUG] [app] Ending request: urn:request:2c8333e2-d55c-4a50-bd3b-f5511b94504d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c8333e2-d55c-4a50-bd3b-f5511b94504d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:59,240 [246] [DEBUG] [app] Ending request: urn:request:ec333582-fa83-4149-86b8-6e094694cac5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec333582-fa83-4149-86b8-6e094694cac5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:28:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:28:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:28:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:59,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:28:59,243 [237] [DEBUG] [app] Starting request: urn:request:fa257c26-c3f7-498e-941a-8b6b103c1ee9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:59,243 [237] [DEBUG] [app] Ending request: urn:request:fa257c26-c3f7-498e-941a-8b6b103c1ee9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa257c26-c3f7-498e-941a-8b6b103c1ee9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:28:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:28:59,244 [235] [DEBUG] [app] Starting request: urn:request:313d9a23-6447-41c1-a570-f2c54f91c15f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:28:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:59,244 [235] [DEBUG] [app] Ending request: urn:request:313d9a23-6447-41c1-a570-f2c54f91c15f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:313d9a23-6447-41c1-a570-f2c54f91c15f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:28:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:59,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:28:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:28:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:28:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:28:59,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:28:59,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:28:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:59,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:28:59,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:28:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:28:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:59,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:28:59,255 [237] [DEBUG] [app] Ending request: urn:request:63cbd0fc-d507-4afb-aa54-c5e163712316 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63cbd0fc-d507-4afb-aa54-c5e163712316', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:28:59,256 [238] [DEBUG] [app] Ending request: urn:request:d991a080-7129-4444-bc0f-a4276eddae53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d991a080-7129-4444-bc0f-a4276eddae53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:28:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:28:59,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:28:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:28:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
exportactionlogsworker stdout | 2025-02-07 03:28:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:28:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:04.936688+00:00 (in 4.998271 seconds)
exportactionlogsworker stdout | 2025-02-07 03:28:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:59 UTC)" (scheduled at 2025-02-07 03:28:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:28:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:28:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:29:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:29:02,386 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:29:03,402 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
repositorygcworker stdout | 2025-02-07 03:29:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:29:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:23.420886+00:00 (in 20.001301 seconds)
repositorygcworker stdout | 2025-02-07 03:29:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:03 UTC)" (scheduled at 2025-02-07 03:29:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:29:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:29:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 3, 419871), True, datetime.datetime(2025, 2, 7, 3, 29, 3, 419871), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:29:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:29:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:29:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:29:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:29:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:33.919286+00:00 (in 29.999513 seconds)
buildlogsarchiver stdout | 2025-02-07 03:29:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:33 UTC)" (scheduled at 2025-02-07 03:29:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:29:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 29, 3, 920059), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:29:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:29:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:29:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:29:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:29:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:52.937648+00:00 (in 48.000532 seconds)
exportactionlogsworker stdout | 2025-02-07 03:29:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:04 UTC)" (scheduled at 2025-02-07 03:29:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:29:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:29:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 4, 937414), True, datetime.datetime(2025, 2, 7, 3, 29, 4, 937414), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:29:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:29:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:29:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:29:08,018 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:29:08,560 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:29:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:29:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:18.732147+00:00 (in 9.999436 seconds)
notificationworker stdout | 2025-02-07 03:29:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:18 UTC)" (scheduled at 2025-02-07 03:29:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:29:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:29:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 8, 733087), True, datetime.datetime(2025, 2, 7, 3, 29, 8, 733087), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:29:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:29:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:29:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:29:09,818 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:29:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:29:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:24.423055+00:00 (in 13.998360 seconds)
gcworker stdout | 2025-02-07 03:29:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:40 UTC)" (scheduled at 2025-02-07 03:29:10.424250+00:00)
gcworker stdout | 2025-02-07 03:29:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:29:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898650431, None, 1, 0])
gcworker stdout | 2025-02-07 03:29:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:29:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:29:10,618 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:29:10,705 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:29:11,773 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:29:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:29:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:41.831521+00:00 (in 29.999567 seconds)
autopruneworker stdout | 2025-02-07 03:29:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:41 UTC)" (scheduled at 2025-02-07 03:29:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:29:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895351836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:29:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:29:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:29:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:41 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:12,087 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:29:12,811 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:29:12,869 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:29:12,926 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:29:13,302 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:29:13,775 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:29:13,853 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:29:13,900 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:29:14,235 [238] [DEBUG] [app] Starting request: urn:request:e7bc46b0-9ad3-4d81-b242-417b8969e74f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:14,235 [235] [DEBUG] [app] Starting request: urn:request:c4b87b22-e5c2-4155-923d-4b48ae7cc245 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:29:14,238 [245] [DEBUG] [app] Starting request: urn:request:8a767c27-5cb8-41d2-9b8c-094168aa7a01 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:14,238 [246] [DEBUG] [app] Starting request: urn:request:2dca0c3d-2b5e-4778-a796-b8023d7c79c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:14,238 [245] [DEBUG] [app] Ending request: urn:request:8a767c27-5cb8-41d2-9b8c-094168aa7a01 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a767c27-5cb8-41d2-9b8c-094168aa7a01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:29:14,238 [246] [DEBUG] [app] Ending request: urn:request:2dca0c3d-2b5e-4778-a796-b8023d7c79c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2dca0c3d-2b5e-4778-a796-b8023d7c79c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:29:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:29:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:29:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:14,241 [237] [DEBUG] [app] Starting request: urn:request:d8f14db7-9b58-417f-88b8-c8e3c919256f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:14,241 [237] [DEBUG] [app] Ending request: urn:request:d8f14db7-9b58-417f-88b8-c8e3c919256f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8f14db7-9b58-417f-88b8-c8e3c919256f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:29:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:14,242 [235] [DEBUG] [app] Starting request: urn:request:333b3d0a-036d-4ed5-a767-57c416dd9d9b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:14,242 [235] [DEBUG] [app] Ending request: urn:request:333b3d0a-036d-4ed5-a767-57c416dd9d9b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:333b3d0a-036d-4ed5-a767-57c416dd9d9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:29:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:29:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:29:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:29:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:14,254 [238] [DEBUG] [app] Ending request: urn:request:e7bc46b0-9ad3-4d81-b242-417b8969e74f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e7bc46b0-9ad3-4d81-b242-417b8969e74f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:29:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:29:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:29:14,254 [235] [DEBUG] [app] Ending request: urn:request:c4b87b22-e5c2-4155-923d-4b48ae7cc245 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4b87b22-e5c2-4155-923d-4b48ae7cc245', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:29:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:29:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:29:15,538 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:29:15,756 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:29:15,759 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:29:15,763 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:29:15,766 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:29:15,769 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:29:15,932 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:29:16,207 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:29:16,784 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:29:16,788 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:29:16,791 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:29:17,719 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:29:17,724 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:29:17,731 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:29:17,734 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:29:17,739 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:29:17,742 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:29:17,745 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:29:17,748 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:29:17,750 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:29:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:29:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:28.732147+00:00 (in 9.999521 seconds)
notificationworker stdout | 2025-02-07 03:29:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:28 UTC)" (scheduled at 2025-02-07 03:29:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:29:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:29:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 18, 732939), True, datetime.datetime(2025, 2, 7, 3, 29, 18, 732939), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:29:18,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:29:18,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:29:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:29:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:29:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:03.419110+00:00 (in 39.997749 seconds)
repositorygcworker stdout | 2025-02-07 03:29:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:23 UTC)" (scheduled at 2025-02-07 03:29:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:29:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:29:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:29:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:29:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:40.424250+00:00 (in 16.000674 seconds)
gcworker stdout | 2025-02-07 03:29:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:54 UTC)" (scheduled at 2025-02-07 03:29:24.423055+00:00)
gcworker stdout | 2025-02-07 03:29:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:29:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:29:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:29:25,114 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:29:27,110 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:29:28,104 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:29:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:29:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:38.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 03:29:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:38 UTC)" (scheduled at 2025-02-07 03:29:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:29:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:29:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 28, 732919), True, datetime.datetime(2025, 2, 7, 3, 29, 28, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:29:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:29:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:29:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:29:29,236 [238] [DEBUG] [app] Starting request: urn:request:982d6c8b-fe84-4971-bc4d-b1e03b40af15 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:29,236 [236] [DEBUG] [app] Starting request: urn:request:be68a839-d5f2-404d-82af-ebe3690f1ff7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:29:29,239 [246] [DEBUG] [app] Starting request: urn:request:a9e84ef9-d004-4baa-88eb-674347f9bceb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:29,239 [239] [DEBUG] [app] Starting request: urn:request:d6def960-1668-4878-bdc9-de4e45779573 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:29,239 [246] [DEBUG] [app] Ending request: urn:request:a9e84ef9-d004-4baa-88eb-674347f9bceb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9e84ef9-d004-4baa-88eb-674347f9bceb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:29:29,239 [239] [DEBUG] [app] Ending request: urn:request:d6def960-1668-4878-bdc9-de4e45779573 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6def960-1668-4878-bdc9-de4e45779573', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:29:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:29:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:29,242 [236] [DEBUG] [app] Starting request: urn:request:4b429025-4476-4049-8e55-b699ee15dcf8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:29,242 [236] [DEBUG] [app] Ending request: urn:request:4b429025-4476-4049-8e55-b699ee15dcf8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b429025-4476-4049-8e55-b699ee15dcf8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:29:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:29,243 [235] [DEBUG] [app] Starting request: urn:request:5cbb113e-cfb6-42b5-a045-a92a269b5dbd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:29,243 [235] [DEBUG] [app] Ending request: urn:request:5cbb113e-cfb6-42b5-a045-a92a269b5dbd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5cbb113e-cfb6-42b5-a045-a92a269b5dbd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:29:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:29:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:29:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:29,255 [238] [DEBUG] [app] Ending request: urn:request:982d6c8b-fe84-4971-bc4d-b1e03b40af15 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:982d6c8b-fe84-4971-bc4d-b1e03b40af15', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:29:29,255 [236] [DEBUG] [app] Ending request: urn:request:be68a839-d5f2-404d-82af-ebe3690f1ff7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:be68a839-d5f2-404d-82af-ebe3690f1ff7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:29:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:29:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:29:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:29:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:29:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:29:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:32.625945+00:00 (in 1.004359 seconds)
namespacegcworker stdout | 2025-02-07 03:29:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:31 UTC)" (scheduled at 2025-02-07 03:29:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:29:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:29:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 31, 621887), True, datetime.datetime(2025, 2, 7, 3, 29, 31, 621887), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:29:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:29:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:29:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:29:32,417 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:29:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:29:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:31.621131+00:00 (in 58.994753 seconds)
namespacegcworker stdout | 2025-02-07 03:29:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:32 UTC)" (scheduled at 2025-02-07 03:29:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:29:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:29:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:29:33,439 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:29:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:29:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:03.919286+00:00 (in 29.999476 seconds)
buildlogsarchiver stdout | 2025-02-07 03:29:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:03 UTC)" (scheduled at 2025-02-07 03:29:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:29:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 29, 33, 920071), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:29:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:29:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:29:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:56.713547+00:00 (in 18.997159 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:37 UTC)" (scheduled at 2025-02-07 03:29:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:29:38,054 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:29:38,596 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:29:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:29:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:48.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 03:29:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:48 UTC)" (scheduled at 2025-02-07 03:29:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:29:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:29:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 38, 732902), True, datetime.datetime(2025, 2, 7, 3, 29, 38, 732902), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:29:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:29:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:29:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:29:39,840 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:29:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:29:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:54.423055+00:00 (in 13.998252 seconds)
gcworker stdout | 2025-02-07 03:29:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:10 UTC)" (scheduled at 2025-02-07 03:29:40.424250+00:00)
gcworker stdout | 2025-02-07 03:29:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:29:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898680431, None, 1, 0])
gcworker stdout | 2025-02-07 03:29:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:29:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:29:40,651 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:29:40,718 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:29:41,809 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
autopruneworker stdout | 2025-02-07 03:29:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:29:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:11.831521+00:00 (in 29.999537 seconds)
autopruneworker stdout | 2025-02-07 03:29:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:11 UTC)" (scheduled at 2025-02-07 03:29:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:29:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895381836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:29:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:29:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:29:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:11 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:42,112 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:29:42,842 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:29:42,901 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:29:42,937 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:29:43,314 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:29:43,811 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:29:43,889 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:29:43,932 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:29:44,235 [238] [DEBUG] [app] Starting request: urn:request:abdf1e85-005f-42a4-bd9f-dc5d371a80aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:44,235 [235] [DEBUG] [app] Starting request: urn:request:ffaa2257-b597-4f8c-b05e-c4e741a31559 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:29:44,238 [245] [DEBUG] [app] Starting request: urn:request:834806d1-b858-4fce-a9fe-0d26e78446c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:44,238 [245] [DEBUG] [app] Ending request: urn:request:834806d1-b858-4fce-a9fe-0d26e78446c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:834806d1-b858-4fce-a9fe-0d26e78446c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:29:44,238 [244] [DEBUG] [app] Starting request: urn:request:a1c9e696-5bd5-4ca1-9935-c74e4b0e4fb6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:44,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:29:44,238 [244] [DEBUG] [app] Ending request: urn:request:a1c9e696-5bd5-4ca1-9935-c74e4b0e4fb6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a1c9e696-5bd5-4ca1-9935-c74e4b0e4fb6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:29:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:29:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:29:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:44,241 [237] [DEBUG] [app] Starting request: urn:request:ab410e8b-6f73-404d-b9f1-f051aecb5522 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:44,242 [237] [DEBUG] [app] Ending request: urn:request:ab410e8b-6f73-404d-b9f1-f051aecb5522 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ab410e8b-6f73-404d-b9f1-f051aecb5522', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:29:44,242 [235] [DEBUG] [app] Starting request: urn:request:b341cfb4-062d-4eb9-afb4-93a90e1417fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:29:44,242 [235] [DEBUG] [app] Ending request: urn:request:b341cfb4-062d-4eb9-afb4-93a90e1417fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b341cfb4-062d-4eb9-afb4-93a90e1417fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:29:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:29:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:29:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:29:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:44,254 [235] [DEBUG] [app] Ending request: urn:request:ffaa2257-b597-4f8c-b05e-c4e741a31559 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ffaa2257-b597-4f8c-b05e-c4e741a31559', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:29:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:29:44,254 [238] [DEBUG] [app] Ending request: urn:request:abdf1e85-005f-42a4-bd9f-dc5d371a80aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:abdf1e85-005f-42a4-bd9f-dc5d371a80aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:29:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:29:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:29:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:29:45,574 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:29:45,764 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:29:45,768 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:29:45,771 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:29:45,774 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:29:45,777 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:29:45,968 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:29:46,234 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:29:46,793 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:29:46,795 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:29:46,798 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:29:47,730 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:29:47,734 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:29:47,741 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:29:47,743 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:29:47,749 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:29:47,752 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:29:47,755 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:29:47,759 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:29:47,761 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:29:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:29:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:50.814677+00:00 (in 2.082083 seconds)
notificationworker stdout | 2025-02-07 03:29:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:58 UTC)" (scheduled at 2025-02-07 03:29:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:29:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:29:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 48, 732874), True, datetime.datetime(2025, 2, 7, 3, 29, 48, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:29:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:29:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:29:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:29:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:29:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:29:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:58.732147+00:00 (in 7.917024 seconds)
notificationworker stdout | 2025-02-07 03:29:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:50 UTC)" (scheduled at 2025-02-07 03:29:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:29:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:29:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:29:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:29:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:29:59.937899+00:00 (in 6.999810 seconds)
exportactionlogsworker stdout | 2025-02-07 03:29:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:34:52 UTC)" (scheduled at 2025-02-07 03:29:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:29:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 29, 52, 938354), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:29:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 29, 52, 938354), True, datetime.datetime(2025, 2, 7, 3, 29, 52, 938354), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:29:52,946 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 29, 52, 938354), True, datetime.datetime(2025, 2, 7, 3, 29, 52, 938354), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 29, 52, 938354), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:29:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:29:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:34:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:29:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:29:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:10.424250+00:00 (in 16.000707 seconds)
gcworker stdout | 2025-02-07 03:29:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:24 UTC)" (scheduled at 2025-02-07 03:29:54.423055+00:00)
gcworker stdout | 2025-02-07 03:29:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:29:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:29:55,151 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:29:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:29:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:56.118855+00:00 (in 59.999488 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:29:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:56 UTC)" (scheduled at 2025-02-07 03:29:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:29:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:29:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:29:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:37.715878+00:00 (in 41.001858 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:56 UTC)" (scheduled at 2025-02-07 03:29:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:29:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:29:57,120 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:29:58,140 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:29:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:29:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:08.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 03:29:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:08 UTC)" (scheduled at 2025-02-07 03:29:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:29:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:29:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 29, 58, 732803), True, datetime.datetime(2025, 2, 7, 3, 29, 58, 732803), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:29:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:29:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:29:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:29:59,235 [235] [DEBUG] [app] Starting request: urn:request:b53be2bf-e0c8-44ed-82fe-ab5fb66bf4c5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:59,235 [237] [DEBUG] [app] Starting request: urn:request:88deea53-b892-4afb-b420-90a674902db1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:29:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:29:59,238 [246] [DEBUG] [app] Starting request: urn:request:2d2e795e-3f19-4050-8838-d03deada80ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:59,238 [239] [DEBUG] [app] Starting request: urn:request:61644066-e8c2-4649-84a0-80dc88a5507f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:29:59,238 [246] [DEBUG] [app] Ending request: urn:request:2d2e795e-3f19-4050-8838-d03deada80ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d2e795e-3f19-4050-8838-d03deada80ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:29:59,238 [239] [DEBUG] [app] Ending request: urn:request:61644066-e8c2-4649-84a0-80dc88a5507f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:61644066-e8c2-4649-84a0-80dc88a5507f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:29:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:29:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:29:59,242 [236] [DEBUG] [app] Starting request: urn:request:1cfd4e56-b25e-4e63-a923-e73b95ff5e0a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:59,242 [238] [DEBUG] [app] Starting request: urn:request:84e7f6ff-3da5-4d8a-8d3c-92708036e530 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:29:59,243 [238] [DEBUG] [app] Ending request: urn:request:84e7f6ff-3da5-4d8a-8d3c-92708036e530 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:84e7f6ff-3da5-4d8a-8d3c-92708036e530', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:29:59,243 [236] [DEBUG] [app] Ending request: urn:request:1cfd4e56-b25e-4e63-a923-e73b95ff5e0a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1cfd4e56-b25e-4e63-a923-e73b95ff5e0a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:29:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:29:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:29:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:29:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:29:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:29:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:29:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:29:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:29:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:29:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:29:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:29:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:29:59,255 [235] [DEBUG] [app] Ending request: urn:request:b53be2bf-e0c8-44ed-82fe-ab5fb66bf4c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b53be2bf-e0c8-44ed-82fe-ab5fb66bf4c5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:29:59,255 [237] [DEBUG] [app] Ending request: urn:request:88deea53-b892-4afb-b420-90a674902db1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:88deea53-b892-4afb-b420-90a674902db1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:29:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:29:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:29:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:29:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:29:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:29:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
exportactionlogsworker stdout | 2025-02-07 03:29:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:29:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:04.936688+00:00 (in 4.998319 seconds)
exportactionlogsworker stdout | 2025-02-07 03:29:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:59 UTC)" (scheduled at 2025-02-07 03:29:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:29:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:29:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:30:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:30:02,447 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:30:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:30:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:23.420886+00:00 (in 20.001309 seconds)
repositorygcworker stdout | 2025-02-07 03:30:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:03 UTC)" (scheduled at 2025-02-07 03:30:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:30:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:30:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 3, 419792), True, datetime.datetime(2025, 2, 7, 3, 30, 3, 419792), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:30:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:30:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:30:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:30:03,471 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:30:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:30:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:33.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 03:30:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:33 UTC)" (scheduled at 2025-02-07 03:30:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:30:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 30, 3, 919998), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:30:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:30:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:30:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:30:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:30:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:59.937899+00:00 (in 55.000753 seconds)
exportactionlogsworker stdout | 2025-02-07 03:30:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:04 UTC)" (scheduled at 2025-02-07 03:30:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:30:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:30:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 4, 937450), True, datetime.datetime(2025, 2, 7, 3, 30, 4, 937450), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:30:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:30:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:30:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:30:08,090 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:30:08,632 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:30:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:30:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:18.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 03:30:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:18 UTC)" (scheduled at 2025-02-07 03:30:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:30:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:30:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 8, 732906), True, datetime.datetime(2025, 2, 7, 3, 30, 8, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:30:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:30:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:30:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:30:09,876 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:30:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:30:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:24.423055+00:00 (in 13.998376 seconds)
gcworker stdout | 2025-02-07 03:30:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:40 UTC)" (scheduled at 2025-02-07 03:30:10.424250+00:00)
gcworker stdout | 2025-02-07 03:30:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:30:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898710431, None, 1, 0])
gcworker stdout | 2025-02-07 03:30:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:30:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:30:10,683 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:30:10,740 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:30:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:30:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:41.831521+00:00 (in 29.999539 seconds)
autopruneworker stdout | 2025-02-07 03:30:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:41 UTC)" (scheduled at 2025-02-07 03:30:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:30:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895411836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:30:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:30:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:30:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:30:11,845 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:12,141 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:30:12,875 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:30:12,935 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:30:12,943 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:30:13,332 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:30:13,848 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:30:13,918 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:30:13,958 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:30:14,235 [235] [DEBUG] [app] Starting request: urn:request:9c41a22d-8755-46c4-a6e0-dfe16cf68bc2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:14,235 [237] [DEBUG] [app] Starting request: urn:request:950c2e21-7ada-44c4-a307-a9634584836b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:30:14,238 [245] [DEBUG] [app] Starting request: urn:request:2ae63906-0498-4d94-a808-0f05acf4c72d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:14,238 [239] [DEBUG] [app] Starting request: urn:request:b3d25dec-7224-4701-99ac-466e9d3bf273 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:14,239 [239] [DEBUG] [app] Ending request: urn:request:b3d25dec-7224-4701-99ac-466e9d3bf273 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b3d25dec-7224-4701-99ac-466e9d3bf273', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:30:14,239 [245] [DEBUG] [app] Ending request: urn:request:2ae63906-0498-4d94-a808-0f05acf4c72d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ae63906-0498-4d94-a808-0f05acf4c72d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:30:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:30:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:30:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:14,242 [237] [DEBUG] [app] Starting request: urn:request:5d2ffc06-0e07-4d2b-92ac-e66a1d669029 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:14,242 [235] [DEBUG] [app] Starting request: urn:request:ade0b7e7-dad8-483f-adf7-961658148d7f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:14,242 [235] [DEBUG] [app] Ending request: urn:request:ade0b7e7-dad8-483f-adf7-961658148d7f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ade0b7e7-dad8-483f-adf7-961658148d7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:14,242 [237] [DEBUG] [app] Ending request: urn:request:5d2ffc06-0e07-4d2b-92ac-e66a1d669029 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5d2ffc06-0e07-4d2b-92ac-e66a1d669029', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:30:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:30:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:14,253 [237] [DEBUG] [app] Ending request: urn:request:950c2e21-7ada-44c4-a307-a9634584836b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:950c2e21-7ada-44c4-a307-a9634584836b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:14,253 [235] [DEBUG] [app] Ending request: urn:request:9c41a22d-8755-46c4-a6e0-dfe16cf68bc2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c41a22d-8755-46c4-a6e0-dfe16cf68bc2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:30:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 03:30:15,606 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:30:15,772 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:30:15,776 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:30:15,779 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:30:15,782 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:30:15,785 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:30:16,004 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:30:16,258 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:30:16,801 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:30:16,805 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:30:16,807 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:30:17,743 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:30:17,746 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:30:17,754 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:30:17,756 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:30:17,761 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:30:17,764 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:30:17,767 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:30:17,770 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:30:17,773 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:30:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:30:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 03:30:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:28 UTC)" (scheduled at 2025-02-07 03:30:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:30:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:30:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 18, 732791), True, datetime.datetime(2025, 2, 7, 3, 30, 18, 732791), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:30:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:30:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:30:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:30:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:30:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:03.419110+00:00 (in 39.997711 seconds)
repositorygcworker stdout | 2025-02-07 03:30:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:23 UTC)" (scheduled at 2025-02-07 03:30:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:30:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:30:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:30:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:30:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 03:30:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:54 UTC)" (scheduled at 2025-02-07 03:30:24.423055+00:00)
gcworker stdout | 2025-02-07 03:30:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:30:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:30:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:30:25,187 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:30:27,144 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:30:28,177 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:30:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:30:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:38.732147+00:00 (in 9.999529 seconds)
notificationworker stdout | 2025-02-07 03:30:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:38 UTC)" (scheduled at 2025-02-07 03:30:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:30:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:30:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 28, 732903), True, datetime.datetime(2025, 2, 7, 3, 30, 28, 732903), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:30:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:30:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:30:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:30:29,235 [235] [DEBUG] [app] Starting request: urn:request:09fbd8fa-f3e2-44da-ba7d-28548859637a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:29,235 [236] [DEBUG] [app] Starting request: urn:request:77919b8f-12d2-4c4e-bf6c-9d31ae88f852 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:30:29,239 [245] [DEBUG] [app] Starting request: urn:request:c64da2d6-dede-4abd-98a8-2a55fb3ec467 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:29,239 [239] [DEBUG] [app] Starting request: urn:request:46ad9438-4803-46d0-b989-5ddf84f2aa78 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:29,239 [245] [DEBUG] [app] Ending request: urn:request:c64da2d6-dede-4abd-98a8-2a55fb3ec467 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c64da2d6-dede-4abd-98a8-2a55fb3ec467', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:30:29,239 [239] [DEBUG] [app] Ending request: urn:request:46ad9438-4803-46d0-b989-5ddf84f2aa78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:46ad9438-4803-46d0-b989-5ddf84f2aa78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:30:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:30:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:30:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:29,242 [238] [DEBUG] [app] Starting request: urn:request:584e44d1-9162-4a7f-af07-fe20a3191dd6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:29,242 [238] [DEBUG] [app] Ending request: urn:request:584e44d1-9162-4a7f-af07-fe20a3191dd6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:584e44d1-9162-4a7f-af07-fe20a3191dd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:30:29,243 [235] [DEBUG] [app] Starting request: urn:request:01cb15e8-9d25-4b7b-808c-3a9652c36591 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:29,243 [235] [DEBUG] [app] Ending request: urn:request:01cb15e8-9d25-4b7b-808c-3a9652c36591 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01cb15e8-9d25-4b7b-808c-3a9652c36591', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:30:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:30:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:29,255 [235] [DEBUG] [app] Ending request: urn:request:09fbd8fa-f3e2-44da-ba7d-28548859637a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09fbd8fa-f3e2-44da-ba7d-28548859637a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:29,255 [236] [DEBUG] [app] Ending request: urn:request:77919b8f-12d2-4c4e-bf6c-9d31ae88f852 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77919b8f-12d2-4c4e-bf6c-9d31ae88f852', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:30:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
namespacegcworker stdout | 2025-02-07 03:30:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:30:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:32.625945+00:00 (in 1.004327 seconds)
namespacegcworker stdout | 2025-02-07 03:30:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:31 UTC)" (scheduled at 2025-02-07 03:30:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:30:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:30:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 31, 621831), True, datetime.datetime(2025, 2, 7, 3, 30, 31, 621831), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:30:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:30:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:30:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:30:32,483 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:30:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:30:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:31.621131+00:00 (in 58.994715 seconds)
namespacegcworker stdout | 2025-02-07 03:30:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:32 UTC)" (scheduled at 2025-02-07 03:30:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:30:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:30:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:30:33,492 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:30:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:30:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:03.919286+00:00 (in 29.999523 seconds)
buildlogsarchiver stdout | 2025-02-07 03:30:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:03 UTC)" (scheduled at 2025-02-07 03:30:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:30:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 30, 33, 920055), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:30:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:30:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:30:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:56.713547+00:00 (in 18.997205 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:37 UTC)" (scheduled at 2025-02-07 03:30:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:30:38,126 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:30:38,670 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:30:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:30:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:48.732147+00:00 (in 9.999424 seconds)
notificationworker stdout | 2025-02-07 03:30:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:48 UTC)" (scheduled at 2025-02-07 03:30:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:30:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:30:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 38, 733012), True, datetime.datetime(2025, 2, 7, 3, 30, 38, 733012), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:30:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:30:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:30:38,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:30:39,901 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:30:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:30:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:54.423055+00:00 (in 13.998375 seconds)
gcworker stdout | 2025-02-07 03:30:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:10 UTC)" (scheduled at 2025-02-07 03:30:40.424250+00:00)
gcworker stdout | 2025-02-07 03:30:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:30:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898740431, None, 1, 0])
gcworker stdout | 2025-02-07 03:30:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:30:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:30:40,719 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:30:40,774 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:30:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:30:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:11.831521+00:00 (in 29.999519 seconds)
autopruneworker stdout | 2025-02-07 03:30:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:11 UTC)" (scheduled at 2025-02-07 03:30:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:30:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895441836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:30:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:30:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:30:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:30:41,882 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:42,177 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:30:42,905 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:30:42,967 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:30:42,978 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:30:43,346 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:30:43,884 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:30:43,955 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:30:43,968 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:30:44,236 [236] [DEBUG] [app] Starting request: urn:request:c50b3d39-1323-4d03-9fdd-98d1489a74b3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:44,236 [238] [DEBUG] [app] Starting request: urn:request:10a19a0c-7917-4570-bca5-0b74d60b11da (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:44,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:30:44,239 [239] [DEBUG] [app] Starting request: urn:request:21f3c01b-67a5-4e8c-9069-8a51990610ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:44,239 [246] [DEBUG] [app] Starting request: urn:request:40291289-1be5-47eb-8714-06a9a4eddbda (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:44,239 [239] [DEBUG] [app] Ending request: urn:request:21f3c01b-67a5-4e8c-9069-8a51990610ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:21f3c01b-67a5-4e8c-9069-8a51990610ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:30:44,240 [246] [DEBUG] [app] Ending request: urn:request:40291289-1be5-47eb-8714-06a9a4eddbda (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40291289-1be5-47eb-8714-06a9a4eddbda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:30:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:30:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:30:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:30:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:44,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:44,243 [235] [DEBUG] [app] Starting request: urn:request:a1c78a3c-1865-4ddd-85c0-c694a4ba119c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:44,243 [235] [DEBUG] [app] Ending request: urn:request:a1c78a3c-1865-4ddd-85c0-c694a4ba119c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1c78a3c-1865-4ddd-85c0-c694a4ba119c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:30:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:30:44,243 [238] [DEBUG] [app] Starting request: urn:request:295b1689-613b-4e26-ad9a-9de6bf945601 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:44,244 [238] [DEBUG] [app] Ending request: urn:request:295b1689-613b-4e26-ad9a-9de6bf945601 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:295b1689-613b-4e26-ad9a-9de6bf945601', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:30:44,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:44,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:44,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:44,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:44,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:44,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:44,255 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:44,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:44,256 [236] [DEBUG] [app] Ending request: urn:request:c50b3d39-1323-4d03-9fdd-98d1489a74b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c50b3d39-1323-4d03-9fdd-98d1489a74b3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:44,256 [238] [DEBUG] [app] Ending request: urn:request:10a19a0c-7917-4570-bca5-0b74d60b11da (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10a19a0c-7917-4570-bca5-0b74d60b11da', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:44,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:44,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:30:44,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
namespacegcworker stdout | 2025-02-07 03:30:45,642 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:30:45,781 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:30:45,784 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:30:45,787 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:30:45,791 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:30:45,793 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:30:46,033 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:30:46,292 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:30:46,809 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:30:46,813 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:30:46,817 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:30:47,752 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:30:47,755 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:30:47,765 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:30:47,768 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:30:47,772 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:30:47,777 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:30:47,780 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:30:47,783 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:30:47,785 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:30:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:30:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:50.814677+00:00 (in 2.082017 seconds)
notificationworker stdout | 2025-02-07 03:30:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:58 UTC)" (scheduled at 2025-02-07 03:30:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:30:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:30:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 48, 732959), True, datetime.datetime(2025, 2, 7, 3, 30, 48, 732959), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:30:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:30:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:30:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:30:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:30:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:30:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:30:58.732147+00:00 (in 7.916999 seconds)
notificationworker stdout | 2025-02-07 03:30:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:50 UTC)" (scheduled at 2025-02-07 03:30:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:30:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:30:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:30:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:30:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:10.424250+00:00 (in 16.000666 seconds)
gcworker stdout | 2025-02-07 03:30:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:24 UTC)" (scheduled at 2025-02-07 03:30:54.423055+00:00)
gcworker stdout | 2025-02-07 03:30:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:30:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:30:55,218 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:30:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:30:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:56.118855+00:00 (in 59.999442 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:30:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:56 UTC)" (scheduled at 2025-02-07 03:30:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:30:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:30:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:30:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:37.715878+00:00 (in 41.001886 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:56 UTC)" (scheduled at 2025-02-07 03:30:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:30:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:30:57,182 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:30:58,213 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:30:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:30:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:08.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 03:30:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:08 UTC)" (scheduled at 2025-02-07 03:30:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:30:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:30:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 30, 58, 732903), True, datetime.datetime(2025, 2, 7, 3, 30, 58, 732903), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:30:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:30:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:30:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:30:59,235 [237] [DEBUG] [app] Starting request: urn:request:9e0d0041-bffe-4097-a633-714bb6914ca5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:59,235 [235] [DEBUG] [app] Starting request: urn:request:eebf5d34-efac-4caf-85f8-24c7d3fc01be (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:30:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:59,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:30:59,239 [239] [DEBUG] [app] Starting request: urn:request:4f75c51e-ce86-40fb-ba23-cb4c4e201484 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:59,240 [239] [DEBUG] [app] Ending request: urn:request:4f75c51e-ce86-40fb-ba23-cb4c4e201484 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f75c51e-ce86-40fb-ba23-cb4c4e201484', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:30:59,240 [245] [DEBUG] [app] Starting request: urn:request:1b0ea777-c8d3-49e0-9d1b-d43d673979d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:30:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:30:59,240 [245] [DEBUG] [app] Ending request: urn:request:1b0ea777-c8d3-49e0-9d1b-d43d673979d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b0ea777-c8d3-49e0-9d1b-d43d673979d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:30:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:30:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:59,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:30:59,243 [238] [DEBUG] [app] Starting request: urn:request:5584d85c-1b04-4c65-a4d3-ee9f17dbb346 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:59,243 [236] [DEBUG] [app] Starting request: urn:request:7222a722-715f-4980-bfa9-0a415211132f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:30:59,243 [238] [DEBUG] [app] Ending request: urn:request:5584d85c-1b04-4c65-a4d3-ee9f17dbb346 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5584d85c-1b04-4c65-a4d3-ee9f17dbb346', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:59,244 [236] [DEBUG] [app] Ending request: urn:request:7222a722-715f-4980-bfa9-0a415211132f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7222a722-715f-4980-bfa9-0a415211132f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:30:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:30:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:30:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:30:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:30:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:30:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:30:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:30:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:30:59,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:59,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:30:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:30:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:30:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:30:59,255 [237] [DEBUG] [app] Ending request: urn:request:9e0d0041-bffe-4097-a633-714bb6914ca5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e0d0041-bffe-4097-a633-714bb6914ca5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:59,256 [235] [DEBUG] [app] Ending request: urn:request:eebf5d34-efac-4caf-85f8-24c7d3fc01be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eebf5d34-efac-4caf-85f8-24c7d3fc01be', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:30:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:30:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:30:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
gunicorn-web stdout | 2025-02-07 03:30:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:30:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:30:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:30:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:04.936688+00:00 (in 4.998287 seconds)
exportactionlogsworker stdout | 2025-02-07 03:30:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:59 UTC)" (scheduled at 2025-02-07 03:30:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:30:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:30:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:31:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:31:02,510 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:31:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:31:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:23.420886+00:00 (in 20.001271 seconds)
repositorygcworker stdout | 2025-02-07 03:31:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:03 UTC)" (scheduled at 2025-02-07 03:31:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:31:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:31:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 3, 419935), True, datetime.datetime(2025, 2, 7, 3, 31, 3, 419935), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:31:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:31:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:31:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:31:03,521 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:31:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:31:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:33.919286+00:00 (in 29.999502 seconds)
buildlogsarchiver stdout | 2025-02-07 03:31:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:33 UTC)" (scheduled at 2025-02-07 03:31:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:31:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 31, 3, 920041), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:31:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:31:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:31:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:31:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:31:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:59.937899+00:00 (in 55.000748 seconds)
exportactionlogsworker stdout | 2025-02-07 03:31:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:04 UTC)" (scheduled at 2025-02-07 03:31:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:31:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:31:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 4, 937465), True, datetime.datetime(2025, 2, 7, 3, 31, 4, 937465), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:31:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:31:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:31:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:04 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,057 [55] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,058 [55] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-02-07 03:31:05 UTC)" (scheduled at 2025-02-07 03:31:05.057762+00:00)
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,058 [55] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,058 [55] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ...
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,059 [55] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:05.057762+00:00 (in 3599.998758 seconds)
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,062 [55] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,062 [55] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,062 [55] [DEBUG] [__main__] Performing blob upload cleanup
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,063 [55] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 5, 3, 31, 5, 62653), 1, 0])
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,069 [55] [DEBUG] [__main__] No additional stale blob uploads found
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,069 [55] [DEBUG] [data.database] Disconnecting from database.
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,070 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/endpoints.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,080 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/sdk-default-configuration.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,080 [55] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fda72c66b80>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,089 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/s3/2006-03-01/service-2.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,104 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,106 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/partitions.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,108 [55] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fda72cc2c10>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,108 [55] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fda6ebe71f0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,135 [55] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fda72cc29d0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,137 [55] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60)
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,139 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/_retry.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,139 [55] [DEBUG] [botocore.client] Registering retry handlers for service: s3
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,139 [55] [DEBUG] [botocore.utils] Registering S3 region redirector handler
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,148 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/boto3/data/s3/2006-03-01/resources-1.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,149 [55] [DEBUG] [botocore.hooks] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fda72c66b80>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,150 [55] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fda72cc2c10>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,150 [55] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fda6ebe71f0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,150 [55] [DEBUG] [botocore.hooks] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fda72cc29d0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,151 [55] [DEBUG] [botocore.endpoint] Setting s3 timeout as (60, 60)
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,153 [55] [DEBUG] [botocore.client] Registering retry handlers for service: s3
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,153 [55] [DEBUG] [botocore.utils] Registering S3 region redirector handler
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,153 [55] [DEBUG] [boto3.resources.factory] Loading s3:s3
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,154 [55] [DEBUG] [boto3.resources.factory] Loading s3:Bucket
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,154 [55] [DEBUG] [boto3.resources.model] Renaming Bucket attribute name
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.hooks] Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7fda6ebe7430>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler <function validate_bucket_name at 0x7fda72c615e0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fda72c4f430>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.HeadBucket: calling handler <function generate_idempotent_uuid at 0x7fda72c61430>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fda72c4f5e0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,155 [55] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.openshift-storage.svc.cluster.local:443', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,156 [55] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,156 [55] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,156 [55] [DEBUG] [botocore.regions] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,156 [55] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler <function add_expect_header at 0x7fda72c61940>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,156 [55] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler <function add_recursion_detection_header at 0x7fda72c610d0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.hooks] Event before-call.s3.HeadBucket: calling handler <function inject_api_version_header_if_needed at 0x7fda72c4bca0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=HeadBucket) with params: {'url_path': '', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360/', 'url': 'https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fda6c1ad490>, 'has_streaming_input': False, 'auth_type': 'v4', 's3_redirect': {'redirected': False, 'bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'params': {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fda6c1ad3a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fda6ddd5b80>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.HeadBucket: calling handler <function set_operation_specific_signer at 0x7fda72c61310>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.hooks] Event before-sign.s3.HeadBucket: calling handler <function remove_arn_from_signing_path at 0x7fda72c4f550>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.auth] Calculating signature using v4 auth.
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.auth] CanonicalRequest:
blobuploadcleanupworker stdout | HEAD
blobuploadcleanupworker stdout | /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | host:s3.openshift-storage.svc.cluster.local
blobuploadcleanupworker stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | x-amz-date:20250207T033105Z
blobuploadcleanupworker stdout | host;x-amz-content-sha256;x-amz-date
blobuploadcleanupworker stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.auth] StringToSign:
blobuploadcleanupworker stdout | AWS4-HMAC-SHA256
blobuploadcleanupworker stdout | 20250207T033105Z
blobuploadcleanupworker stdout | 20250207/us-east-1/s3/aws4_request
blobuploadcleanupworker stdout | 601f738b8ddcbb35e72ddcaa4bd53a05e7ba2419ad212724a0f3696da5a22cff
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.auth] Signature:
blobuploadcleanupworker stdout | 7978d4a12d3c762af4180225302d930891751ab2ac6482324a1f35e1f23ab1b5
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,157 [55] [DEBUG] [botocore.hooks] Event request-created.s3.HeadBucket: calling handler <function add_retry_headers at 0x7fda72c4f3a0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,158 [55] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360, headers={'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20250207T033105Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=pjqBQGfl18IdF9RWc0aJ/20250207/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7978d4a12d3c762af4180225302d930891751ab2ac6482324a1f35e1f23ab1b5', 'amz-sdk-invocation-id': b'1bc309ce-c07d-4c78-89ef-c886d542ad56', 'amz-sdk-request': b'attempt=1'}>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,158 [55] [DEBUG] [botocore.httpsession] Certificate path: /app/lib/python3.9/site-packages/certifi/cacert.pem
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,158 [55] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): s3.openshift-storage.svc.cluster.local:443
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,189 [55] [DEBUG] [urllib3.connectionpool] https://s3.openshift-storage.svc.cluster.local:443 "HEAD /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360 HTTP/1.1" 200 0
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,189 [55] [DEBUG] [botocore.parsers] Response headers: {'Server': 'NooBaa/5.17.3-427bfd2', 'x-amz-request-id': 'm6u7ltuj-b77cwy-q5c', 'x-amz-id-2': 'm6u7ltuj-b77cwy-q5c', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Methods': 'GET,POST,PUT,DELETE,OPTIONS', 'Access-Control-Allow-Headers': 'Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256,amz-sdk-invocation-id,amz-sdk-request', 'Access-Control-Expose-Headers': 'ETag,X-Amz-Version-Id', 'Date': 'Fri, 07 Feb 2025 03:31:05 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5'}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,189 [55] [DEBUG] [botocore.parsers] Response body:
blobuploadcleanupworker stdout | b''
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,190 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda5fb4eac0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,190 [55] [DEBUG] [botocore.retryhandler] No retry needed.
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,190 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.HeadBucket: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,197 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/s3/2006-03-01/paginators-1.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,197 [55] [DEBUG] [botocore.loaders] Loading JSON file: /app/lib/python3.9/site-packages/botocore/data/s3/2006-03-01/paginators-1.sdk-extras.json
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7fda72c4b8b0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7fda72c615e0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fda72c4f430>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7fda72c61430>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fda72c4f5e0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.openshift-storage.svc.cluster.local:443', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.regions] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,198 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7fda72c61940>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function add_recursion_detection_header at 0x7fda72c610d0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7fda72c4bca0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': 'datastorage/registry/uploads', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360/', 'url': 'https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fda6c1ad490>, 'has_streaming_input': False, 'auth_type': 'v4', 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'params': {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'Prefix': 'datastorage/registry/uploads', 'EncodingType': 'url'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fda6c1ad3a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.ListObjectsV2: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fda6ddd5b80>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7fda72c61310>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.hooks] Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7fda72c4f550>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.auth] Calculating signature using v4 auth.
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.auth] CanonicalRequest:
blobuploadcleanupworker stdout | GET
blobuploadcleanupworker stdout | /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | encoding-type=url&list-type=2&prefix=datastorage%2Fregistry%2Fuploads
blobuploadcleanupworker stdout | host:s3.openshift-storage.svc.cluster.local
blobuploadcleanupworker stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | x-amz-date:20250207T033105Z
blobuploadcleanupworker stdout | host;x-amz-content-sha256;x-amz-date
blobuploadcleanupworker stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.auth] StringToSign:
blobuploadcleanupworker stdout | AWS4-HMAC-SHA256
blobuploadcleanupworker stdout | 20250207T033105Z
blobuploadcleanupworker stdout | 20250207/us-east-1/s3/aws4_request
blobuploadcleanupworker stdout | c248f7be12977033cb40d588ff4fc166d99e6f1173b7611ce6c1c3246d4ccb9c
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.auth] Signature:
blobuploadcleanupworker stdout | ac9d7675bc08883a79d52b883571571a3f9649587c3d1731a68e0b6e07543265
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,199 [55] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler <function add_retry_headers at 0x7fda72c4f3a0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,200 [55] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url, headers={'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20250207T033105Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=pjqBQGfl18IdF9RWc0aJ/20250207/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ac9d7675bc08883a79d52b883571571a3f9649587c3d1731a68e0b6e07543265', 'amz-sdk-invocation-id': b'847086ca-112e-42d7-8520-215333e81118', 'amz-sdk-request': b'attempt=1'}>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,200 [55] [DEBUG] [botocore.httpsession] Certificate path: /app/lib/python3.9/site-packages/certifi/cacert.pem
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,205 [55] [DEBUG] [urllib3.connectionpool] https://s3.openshift-storage.svc.cluster.local:443 "GET /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url HTTP/1.1" 200 411
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,205 [55] [DEBUG] [botocore.parsers] Response headers: {'Server': 'NooBaa/5.17.3-427bfd2', 'x-amz-request-id': 'm6u7ltv5-bk7y3l-7pl', 'x-amz-id-2': 'm6u7ltv5-bk7y3l-7pl', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Methods': 'GET,POST,PUT,DELETE,OPTIONS', 'Access-Control-Allow-Headers': 'Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256,amz-sdk-invocation-id,amz-sdk-request', 'Access-Control-Expose-Headers': 'ETag,X-Amz-Version-Id', 'Content-Type': 'application/xml', 'Content-Length': '411', 'Date': 'Fri, 07 Feb 2025 03:31:05 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5'}
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,205 [55] [DEBUG] [botocore.parsers] Response body:
blobuploadcleanupworker stdout | b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360</Name><Prefix>datastorage%2Fregistry%2Fuploads</Prefix><Delimiter>undefined</Delimiter><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><EncodingType>url</EncodingType><StartAfter>undefined</StartAfter><KeyCount>0</KeyCount></ListBucketResult>'
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,206 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda5fb4eac0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,206 [55] [DEBUG] [botocore.retryhandler] No retry needed.
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,206 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,206 [55] [DEBUG] [botocore.hooks] Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7fda72c4b9d0>
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,206 [55] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,206 [55] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,210 [55] [DEBUG] [util.locking] Released lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 03:31:05,210 [55] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-02-07 04:31:05 UTC)" executed successfully
builder stdout | 2025-02-07 03:31:08,163 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:31:08,707 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:31:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:31:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:18.732147+00:00 (in 9.999539 seconds)
notificationworker stdout | 2025-02-07 03:31:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:18 UTC)" (scheduled at 2025-02-07 03:31:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:31:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:31:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 8, 732888), True, datetime.datetime(2025, 2, 7, 3, 31, 8, 732888), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:31:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:31:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:31:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:18 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:31:09,919 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:31:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:31:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:24.423055+00:00 (in 13.998302 seconds)
gcworker stdout | 2025-02-07 03:31:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:40 UTC)" (scheduled at 2025-02-07 03:31:10.424250+00:00)
gcworker stdout | 2025-02-07 03:31:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:31:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898770431, None, 1, 0])
gcworker stdout | 2025-02-07 03:31:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:31:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:31:10,756 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:31:10,807 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:31:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:31:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:41.831521+00:00 (in 29.999535 seconds)
autopruneworker stdout | 2025-02-07 03:31:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:41 UTC)" (scheduled at 2025-02-07 03:31:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:31:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895471836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:31:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:31:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:31:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:31:11,907 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:12,211 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:31:12,925 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:31:12,988 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:31:13,014 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:31:13,383 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:31:13,917 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:31:13,963 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:31:13,985 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:31:14,235 [235] [DEBUG] [app] Starting request: urn:request:8c7c052d-594d-4f02-a4a3-24b33bbb530d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:14,235 [236] [DEBUG] [app] Starting request: urn:request:6a43255e-c0a6-4686-b23d-847bffc5476b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:31:14,238 [244] [DEBUG] [app] Starting request: urn:request:b439d579-04c6-4da0-ac99-e19488cd0162 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:14,238 [246] [DEBUG] [app] Starting request: urn:request:ed0ce15f-bd4d-405f-acef-bfff59b9a533 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:14,239 [244] [DEBUG] [app] Ending request: urn:request:b439d579-04c6-4da0-ac99-e19488cd0162 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b439d579-04c6-4da0-ac99-e19488cd0162', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:31:14,239 [246] [DEBUG] [app] Ending request: urn:request:ed0ce15f-bd4d-405f-acef-bfff59b9a533 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ed0ce15f-bd4d-405f-acef-bfff59b9a533', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:31:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:31:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:14,242 [236] [DEBUG] [app] Starting request: urn:request:aeaf5d36-5105-4c91-9228-43c0208ee933 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:14,242 [236] [DEBUG] [app] Ending request: urn:request:aeaf5d36-5105-4c91-9228-43c0208ee933 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aeaf5d36-5105-4c91-9228-43c0208ee933', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:31:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:14,243 [238] [DEBUG] [app] Starting request: urn:request:3318e475-1725-41f5-9a74-1678167502bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:14,243 [238] [DEBUG] [app] Ending request: urn:request:3318e475-1725-41f5-9a74-1678167502bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3318e475-1725-41f5-9a74-1678167502bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:31:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:31:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:31:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:31:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:14,254 [236] [DEBUG] [app] Ending request: urn:request:6a43255e-c0a6-4686-b23d-847bffc5476b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a43255e-c0a6-4686-b23d-847bffc5476b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:31:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:31:14,254 [235] [DEBUG] [app] Ending request: urn:request:8c7c052d-594d-4f02-a4a3-24b33bbb530d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8c7c052d-594d-4f02-a4a3-24b33bbb530d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:31:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:31:15,680 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:31:15,792 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:31:15,795 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:31:15,799 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:31:15,802 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:31:15,805 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:31:16,069 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:31:16,328 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:31:16,817 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:31:16,821 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:31:16,823 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:31:17,762 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:31:17,765 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:31:17,775 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:31:17,778 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:31:17,780 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:31:17,785 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:31:17,788 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:31:17,792 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:31:17,794 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:31:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:31:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:28.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:31:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:28 UTC)" (scheduled at 2025-02-07 03:31:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:31:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:31:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 18, 732868), True, datetime.datetime(2025, 2, 7, 3, 31, 18, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:31:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:31:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:31:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:31:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:31:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:03.419110+00:00 (in 39.997701 seconds)
repositorygcworker stdout | 2025-02-07 03:31:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:23 UTC)" (scheduled at 2025-02-07 03:31:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:31:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:31:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:31:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:31:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:40.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 03:31:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:54 UTC)" (scheduled at 2025-02-07 03:31:24.423055+00:00)
gcworker stdout | 2025-02-07 03:31:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:31:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:31:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:31:25,228 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:31:27,206 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:31:28,229 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:31:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:31:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:38.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:31:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:38 UTC)" (scheduled at 2025-02-07 03:31:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:31:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:31:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 28, 732862), True, datetime.datetime(2025, 2, 7, 3, 31, 28, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:31:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:31:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:31:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:31:29,235 [238] [DEBUG] [app] Starting request: urn:request:51e48d2c-aefd-4fb4-8470-9cd16d78bad5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:29,235 [235] [DEBUG] [app] Starting request: urn:request:b69c07c8-4fc1-4960-92c5-9d1b5e601905 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:31:29,238 [246] [DEBUG] [app] Starting request: urn:request:a4efd8f4-4ffc-4ee4-be16-23e8ee4cc627 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:29,238 [239] [DEBUG] [app] Starting request: urn:request:df982661-2cec-4bc4-94e0-9d0dd6c72147 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:29,239 [239] [DEBUG] [app] Ending request: urn:request:df982661-2cec-4bc4-94e0-9d0dd6c72147 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df982661-2cec-4bc4-94e0-9d0dd6c72147', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:31:29,239 [246] [DEBUG] [app] Ending request: urn:request:a4efd8f4-4ffc-4ee4-be16-23e8ee4cc627 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4efd8f4-4ffc-4ee4-be16-23e8ee4cc627', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:31:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:31:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:31:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:29,242 [237] [DEBUG] [app] Starting request: urn:request:05efdfbe-d5be-470d-9f0b-e9801355e81b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:29,242 [238] [DEBUG] [app] Starting request: urn:request:7f2f396d-e8c8-411f-8e7e-57e7f50c0790 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:29,242 [238] [DEBUG] [app] Ending request: urn:request:7f2f396d-e8c8-411f-8e7e-57e7f50c0790 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f2f396d-e8c8-411f-8e7e-57e7f50c0790', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:31:29,242 [237] [DEBUG] [app] Ending request: urn:request:05efdfbe-d5be-470d-9f0b-e9801355e81b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05efdfbe-d5be-470d-9f0b-e9801355e81b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:31:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:31:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:31:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:31:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:29,254 [238] [DEBUG] [app] Ending request: urn:request:51e48d2c-aefd-4fb4-8470-9cd16d78bad5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:51e48d2c-aefd-4fb4-8470-9cd16d78bad5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:29,254 [235] [DEBUG] [app] Ending request: urn:request:b69c07c8-4fc1-4960-92c5-9d1b5e601905 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b69c07c8-4fc1-4960-92c5-9d1b5e601905', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:31:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:31:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:31:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
namespacegcworker stdout | 2025-02-07 03:31:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:31:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:32.625945+00:00 (in 1.004335 seconds)
namespacegcworker stdout | 2025-02-07 03:31:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:31 UTC)" (scheduled at 2025-02-07 03:31:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:31:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:31:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 31, 621916), True, datetime.datetime(2025, 2, 7, 3, 31, 31, 621916), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:31:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:31:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:31:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:31:32,538 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:31:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:31:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:13.624301+00:00 (in 40.997849 seconds)
namespacegcworker stdout | 2025-02-07 03:31:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:32 UTC)" (scheduled at 2025-02-07 03:31:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:31:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:31:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:31:33,557 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:31:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:31:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:03.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 03:31:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:03 UTC)" (scheduled at 2025-02-07 03:31:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:31:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 31, 33, 919994), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:31:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:31:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:31:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:56.713547+00:00 (in 18.997169 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:37 UTC)" (scheduled at 2025-02-07 03:31:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:31:38,181 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:31:38,730 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
notificationworker stdout | 2025-02-07 03:31:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:31:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:48.732147+00:00 (in 9.999655 seconds)
notificationworker stdout | 2025-02-07 03:31:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:48 UTC)" (scheduled at 2025-02-07 03:31:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:31:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:31:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 38, 732762), True, datetime.datetime(2025, 2, 7, 3, 31, 38, 732762), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:31:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:31:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:31:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:48 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:31:39,933 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:31:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:31:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:54.423055+00:00 (in 13.998362 seconds)
gcworker stdout | 2025-02-07 03:31:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:10 UTC)" (scheduled at 2025-02-07 03:31:40.424250+00:00)
gcworker stdout | 2025-02-07 03:31:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:31:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898800430, None, 1, 0])
gcworker stdout | 2025-02-07 03:31:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:31:40,432 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:31:40,778 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:31:40,817 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:31:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:31:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:11.831521+00:00 (in 29.999505 seconds)
autopruneworker stdout | 2025-02-07 03:31:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:11 UTC)" (scheduled at 2025-02-07 03:31:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:31:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895501836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:31:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:31:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:31:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:31:41,928 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:42,245 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:31:42,936 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:31:43,025 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:31:43,049 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:31:43,410 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:31:43,936 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:31:44,000 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:31:44,021 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:31:44,236 [235] [DEBUG] [app] Starting request: urn:request:f97e018a-e308-471e-997e-8ecbd38e3b3c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:44,236 [236] [DEBUG] [app] Starting request: urn:request:0e51e551-a454-4a20-97cf-6be99a0f4c57 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:31:44,239 [244] [DEBUG] [app] Starting request: urn:request:23ccc614-e200-4393-8476-d4647a1725d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:44,239 [239] [DEBUG] [app] Starting request: urn:request:04b8fb62-ca18-4d31-9186-878013025701 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:44,239 [239] [DEBUG] [app] Ending request: urn:request:04b8fb62-ca18-4d31-9186-878013025701 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04b8fb62-ca18-4d31-9186-878013025701', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:31:44,239 [244] [DEBUG] [app] Ending request: urn:request:23ccc614-e200-4393-8476-d4647a1725d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23ccc614-e200-4393-8476-d4647a1725d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:31:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:31:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:44,242 [236] [DEBUG] [app] Starting request: urn:request:7864f355-6c99-4160-b135-4942c1aac378 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:44,242 [236] [DEBUG] [app] Ending request: urn:request:7864f355-6c99-4160-b135-4942c1aac378 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7864f355-6c99-4160-b135-4942c1aac378', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:31:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:44,243 [238] [DEBUG] [app] Starting request: urn:request:9f5be965-1624-4db8-a9be-6fa50baf2c7f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:44,243 [238] [DEBUG] [app] Ending request: urn:request:9f5be965-1624-4db8-a9be-6fa50baf2c7f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f5be965-1624-4db8-a9be-6fa50baf2c7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:31:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:31:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:31:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:44,255 [236] [DEBUG] [app] Ending request: urn:request:0e51e551-a454-4a20-97cf-6be99a0f4c57 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e51e551-a454-4a20-97cf-6be99a0f4c57', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:31:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:31:44,255 [235] [DEBUG] [app] Ending request: urn:request:f97e018a-e308-471e-997e-8ecbd38e3b3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f97e018a-e308-471e-997e-8ecbd38e3b3c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:31:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:31:45,716 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:31:45,799 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:31:45,802 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:31:45,805 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:31:45,809 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:31:45,811 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:31:46,095 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:31:46,347 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:31:46,826 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:31:46,829 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:31:46,832 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:31:47,771 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:31:47,774 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:31:47,784 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:31:47,787 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:31:47,789 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:31:47,792 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:31:47,799 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:31:47,802 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:31:47,804 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:31:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:31:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:50.814677+00:00 (in 2.082097 seconds)
notificationworker stdout | 2025-02-07 03:31:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:58 UTC)" (scheduled at 2025-02-07 03:31:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:31:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:31:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 48, 732855), True, datetime.datetime(2025, 2, 7, 3, 31, 48, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:31:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:31:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:31:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:31:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:31:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:31:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:31:58.732147+00:00 (in 7.917016 seconds)
notificationworker stdout | 2025-02-07 03:31:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:50 UTC)" (scheduled at 2025-02-07 03:31:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:31:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:31:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:31:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:31:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:10.424250+00:00 (in 16.000715 seconds)
gcworker stdout | 2025-02-07 03:31:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:24 UTC)" (scheduled at 2025-02-07 03:31:54.423055+00:00)
gcworker stdout | 2025-02-07 03:31:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:31:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:31:55,264 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:31:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:31:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:56.118855+00:00 (in 59.999447 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:31:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:56 UTC)" (scheduled at 2025-02-07 03:31:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:31:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:31:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:31:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:37.715878+00:00 (in 41.001877 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:56 UTC)" (scheduled at 2025-02-07 03:31:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:31:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:31:57,231 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:31:58,250 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:31:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:31:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:08.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 03:31:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:08 UTC)" (scheduled at 2025-02-07 03:31:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:31:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:31:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 31, 58, 732907), True, datetime.datetime(2025, 2, 7, 3, 31, 58, 732907), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:31:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:31:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:31:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:31:59,236 [237] [DEBUG] [app] Starting request: urn:request:44eae8f5-f12d-4fc1-b788-5d11eea3b5e6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:59,236 [238] [DEBUG] [app] Starting request: urn:request:144dd195-f2a1-428d-8f80-19738187b5ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:31:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:31:59,239 [246] [DEBUG] [app] Starting request: urn:request:b4dfc0ed-419b-4979-96f6-d5cab712eaac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:59,239 [239] [DEBUG] [app] Starting request: urn:request:bfe38ef3-de7e-47af-98c6-b5a6c6745ab4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:31:59,239 [239] [DEBUG] [app] Ending request: urn:request:bfe38ef3-de7e-47af-98c6-b5a6c6745ab4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bfe38ef3-de7e-47af-98c6-b5a6c6745ab4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:31:59,239 [246] [DEBUG] [app] Ending request: urn:request:b4dfc0ed-419b-4979-96f6-d5cab712eaac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b4dfc0ed-419b-4979-96f6-d5cab712eaac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:31:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:31:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:31:59,242 [237] [DEBUG] [app] Starting request: urn:request:d3aba4ce-a34d-47b3-aa3e-36298304282a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:59,243 [235] [DEBUG] [app] Starting request: urn:request:df97a050-2769-4beb-9449-d501e20943d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:31:59,243 [237] [DEBUG] [app] Ending request: urn:request:d3aba4ce-a34d-47b3-aa3e-36298304282a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3aba4ce-a34d-47b3-aa3e-36298304282a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:31:59,243 [235] [DEBUG] [app] Ending request: urn:request:df97a050-2769-4beb-9449-d501e20943d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df97a050-2769-4beb-9449-d501e20943d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:31:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:31:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:31:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:31:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:31:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:31:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:31:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:31:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:31:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:31:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:31:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:31:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:31:59,255 [238] [DEBUG] [app] Ending request: urn:request:144dd195-f2a1-428d-8f80-19738187b5ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:144dd195-f2a1-428d-8f80-19738187b5ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:59,255 [237] [DEBUG] [app] Ending request: urn:request:44eae8f5-f12d-4fc1-b788-5d11eea3b5e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44eae8f5-f12d-4fc1-b788-5d11eea3b5e6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:31:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:31:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:31:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:31:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:31:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:31:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:31:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:31:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:04.936688+00:00 (in 4.998276 seconds)
exportactionlogsworker stdout | 2025-02-07 03:31:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:59 UTC)" (scheduled at 2025-02-07 03:31:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:31:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:31:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:32:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:32:02,569 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:32:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:32:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:23.420886+00:00 (in 20.001263 seconds)
repositorygcworker stdout | 2025-02-07 03:32:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:03 UTC)" (scheduled at 2025-02-07 03:32:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:32:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:32:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 3, 419830), True, datetime.datetime(2025, 2, 7, 3, 32, 3, 419830), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:32:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:32:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:32:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:32:03,569 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:32:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:32:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:33.919286+00:00 (in 29.999552 seconds)
buildlogsarchiver stdout | 2025-02-07 03:32:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:33 UTC)" (scheduled at 2025-02-07 03:32:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:32:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 32, 3, 920025), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:32:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:32:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:32:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:32:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:32:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:59.937899+00:00 (in 55.000777 seconds)
exportactionlogsworker stdout | 2025-02-07 03:32:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:04 UTC)" (scheduled at 2025-02-07 03:32:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:32:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:32:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 4, 937416), True, datetime.datetime(2025, 2, 7, 3, 32, 4, 937416), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:32:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:32:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:32:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:32:08,210 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:32:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:18.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 03:32:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:18 UTC)" (scheduled at 2025-02-07 03:32:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:32:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:32:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 8, 732914), True, datetime.datetime(2025, 2, 7, 3, 32, 8, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:32:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:32:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:32:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:32:08,767 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:32:09,969 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:32:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:32:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:24.423055+00:00 (in 13.998354 seconds)
gcworker stdout | 2025-02-07 03:32:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:40 UTC)" (scheduled at 2025-02-07 03:32:10.424250+00:00)
gcworker stdout | 2025-02-07 03:32:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:32:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898830431, None, 1, 0])
gcworker stdout | 2025-02-07 03:32:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:32:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:32:10,814 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:32:10,853 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:32:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:32:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:41.831521+00:00 (in 29.999534 seconds)
autopruneworker stdout | 2025-02-07 03:32:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:41 UTC)" (scheduled at 2025-02-07 03:32:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:32:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895531836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:32:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:32:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:32:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:32:11,952 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:12,281 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:32:12,946 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:32:13,042 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:32:13,086 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:32:13,446 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
namespacegcworker stdout | 2025-02-07 03:32:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:32:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:31.621131+00:00 (in 17.996347 seconds)
namespacegcworker stdout | 2025-02-07 03:32:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:37:13 UTC)" (scheduled at 2025-02-07 03:32:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:32:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 32, 13, 625057), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:32:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 32, 13, 625057), True, datetime.datetime(2025, 2, 7, 3, 32, 13, 625057), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:32:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 32, 13, 625057), True, datetime.datetime(2025, 2, 7, 3, 32, 13, 625057), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 32, 13, 625057), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:32:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:32:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:37:13 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 03:32:13,973 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:32:14,014 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:32:14,030 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:32:14,236 [236] [DEBUG] [app] Starting request: urn:request:935a05ed-ffb2-4563-9af7-c7d4d56f9b32 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:14,236 [238] [DEBUG] [app] Starting request: urn:request:0abaf822-68fd-498f-98ac-55927343b55a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:32:14,240 [239] [DEBUG] [app] Starting request: urn:request:a04d56ef-c368-44c1-8fa6-9440df84e1ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:14,240 [245] [DEBUG] [app] Starting request: urn:request:be224d75-e675-477f-aebe-64649a6c7ef0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:14,240 [245] [DEBUG] [app] Ending request: urn:request:be224d75-e675-477f-aebe-64649a6c7ef0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be224d75-e675-477f-aebe-64649a6c7ef0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:32:14,240 [239] [DEBUG] [app] Ending request: urn:request:a04d56ef-c368-44c1-8fa6-9440df84e1ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a04d56ef-c368-44c1-8fa6-9440df84e1ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:32:14,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:32:14,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:32:14,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:14,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:14,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:14,244 [238] [DEBUG] [app] Starting request: urn:request:eef255d2-702d-4919-9da6-943513352780 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:14,244 [235] [DEBUG] [app] Starting request: urn:request:ad347851-3761-4824-afec-783db74a7be0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:14,244 [238] [DEBUG] [app] Ending request: urn:request:eef255d2-702d-4919-9da6-943513352780 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eef255d2-702d-4919-9da6-943513352780', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:32:14,244 [235] [DEBUG] [app] Ending request: urn:request:ad347851-3761-4824-afec-783db74a7be0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad347851-3761-4824-afec-783db74a7be0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:32:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:32:14,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:32:14,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:14,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:14,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:14,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:14,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:14,250 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:14,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:14,254 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:14,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:14,256 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:14,257 [238] [DEBUG] [app] Ending request: urn:request:0abaf822-68fd-498f-98ac-55927343b55a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0abaf822-68fd-498f-98ac-55927343b55a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:14,257 [236] [DEBUG] [app] Ending request: urn:request:935a05ed-ffb2-4563-9af7-c7d4d56f9b32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:935a05ed-ffb2-4563-9af7-c7d4d56f9b32', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:14,257 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:32:14,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:32:14,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:32:14,258 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
namespacegcworker stdout | 2025-02-07 03:32:15,731 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:32:15,807 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:32:15,810 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:32:15,812 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:32:15,816 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:32:15,818 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:32:16,122 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:32:16,359 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:32:16,837 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:32:16,840 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:32:16,843 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:32:17,779 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:32:17,782 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:32:17,793 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:32:17,796 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:32:17,798 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:32:17,801 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:32:17,806 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:32:17,810 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:32:17,815 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:32:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:28.732147+00:00 (in 9.999512 seconds)
notificationworker stdout | 2025-02-07 03:32:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:28 UTC)" (scheduled at 2025-02-07 03:32:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:32:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:32:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 18, 732965), True, datetime.datetime(2025, 2, 7, 3, 32, 18, 732965), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:32:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:32:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:32:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:32:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:32:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:03.419110+00:00 (in 39.997737 seconds)
repositorygcworker stdout | 2025-02-07 03:32:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:23 UTC)" (scheduled at 2025-02-07 03:32:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:32:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:32:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:32:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:32:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:40.424250+00:00 (in 16.000718 seconds)
gcworker stdout | 2025-02-07 03:32:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:54 UTC)" (scheduled at 2025-02-07 03:32:24.423055+00:00)
gcworker stdout | 2025-02-07 03:32:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:32:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:32:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:32:25,300 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:32:27,268 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:32:28,269 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:32:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:33.814166+00:00 (in 5.081509 seconds)
notificationworker stdout | 2025-02-07 03:32:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:38 UTC)" (scheduled at 2025-02-07 03:32:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:32:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:32:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 28, 732963), True, datetime.datetime(2025, 2, 7, 3, 32, 28, 732963), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:32:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:32:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:32:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:32:29,236 [235] [DEBUG] [app] Starting request: urn:request:e70ca59b-0efd-4176-9531-e5a899a757a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:29,236 [236] [DEBUG] [app] Starting request: urn:request:d4edb953-f25e-47ed-8216-9eb8c640ad41 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:29,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:32:29,240 [239] [DEBUG] [app] Starting request: urn:request:1117fc57-a9b6-4dd0-b35e-650a041fbca0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:29,240 [243] [DEBUG] [app] Starting request: urn:request:78a89ae8-c504-460d-872e-a08c4acc187f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:29,240 [239] [DEBUG] [app] Ending request: urn:request:1117fc57-a9b6-4dd0-b35e-650a041fbca0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1117fc57-a9b6-4dd0-b35e-650a041fbca0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:32:29,240 [243] [DEBUG] [app] Ending request: urn:request:78a89ae8-c504-460d-872e-a08c4acc187f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78a89ae8-c504-460d-872e-a08c4acc187f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:32:29,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:32:29,241 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:32:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:29,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:29,243 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:29,243 [235] [DEBUG] [app] Starting request: urn:request:2692391b-ab83-4bf5-9e46-9711a441485f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:29,244 [235] [DEBUG] [app] Ending request: urn:request:2692391b-ab83-4bf5-9e46-9711a441485f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2692391b-ab83-4bf5-9e46-9711a441485f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:32:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:32:29,244 [238] [DEBUG] [app] Starting request: urn:request:39c7e1d8-be90-469c-9f6a-5b4e0b89c889 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:29,244 [238] [DEBUG] [app] Ending request: urn:request:39c7e1d8-be90-469c-9f6a-5b4e0b89c889 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39c7e1d8-be90-469c-9f6a-5b4e0b89c889', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:32:29,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:32:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:29,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:32:29,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:29,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:29,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:29,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:29,254 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:29,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:29,255 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:29,256 [235] [DEBUG] [app] Ending request: urn:request:e70ca59b-0efd-4176-9531-e5a899a757a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e70ca59b-0efd-4176-9531-e5a899a757a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:29,256 [236] [DEBUG] [app] Ending request: urn:request:d4edb953-f25e-47ed-8216-9eb8c640ad41 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d4edb953-f25e-47ed-8216-9eb8c640ad41', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:29,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:32:29,257 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:32:29,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:32:29,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:32:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:32:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:32.625945+00:00 (in 1.004268 seconds)
namespacegcworker stdout | 2025-02-07 03:32:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:31 UTC)" (scheduled at 2025-02-07 03:32:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:32:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:32:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 31, 621998), True, datetime.datetime(2025, 2, 7, 3, 32, 31, 621998), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:32:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:32:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:32:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:31 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:32:32,605 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
namespacegcworker stdout | 2025-02-07 03:32:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:32:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:31.621131+00:00 (in 58.994653 seconds)
namespacegcworker stdout | 2025-02-07 03:32:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:32 UTC)" (scheduled at 2025-02-07 03:32:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:32:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:32:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:32:33,589 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 03:32:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:38.732147+00:00 (in 4.917511 seconds)
notificationworker stdout | 2025-02-07 03:32:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:37:33 UTC)" (scheduled at 2025-02-07 03:32:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:32:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 32, 33, 814887), 'notification/%'])
notificationworker stdout | 2025-02-07 03:32:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 32, 33, 814887), True, datetime.datetime(2025, 2, 7, 3, 32, 33, 814887), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:32:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 32, 33, 814887), True, datetime.datetime(2025, 2, 7, 3, 32, 33, 814887), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 32, 33, 814887), 'notification/%'])
notificationworker stdout | 2025-02-07 03:32:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:32:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:37:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:32:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:32:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:03.919286+00:00 (in 29.999556 seconds)
buildlogsarchiver stdout | 2025-02-07 03:32:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:03 UTC)" (scheduled at 2025-02-07 03:32:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:32:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 32, 33, 920005), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:32:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:32:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:32:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:56.713547+00:00 (in 18.997182 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:37 UTC)" (scheduled at 2025-02-07 03:32:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:32:38,247 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:32:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:48.732147+00:00 (in 9.999488 seconds)
notificationworker stdout | 2025-02-07 03:32:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:48 UTC)" (scheduled at 2025-02-07 03:32:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:32:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:32:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 38, 732989), True, datetime.datetime(2025, 2, 7, 3, 32, 38, 732989), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:32:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:32:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:32:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:32:38,794 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:32:39,989 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:32:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:32:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:54.423055+00:00 (in 13.998337 seconds)
gcworker stdout | 2025-02-07 03:32:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:10 UTC)" (scheduled at 2025-02-07 03:32:40.424250+00:00)
gcworker stdout | 2025-02-07 03:32:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:32:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898860431, None, 1, 0])
gcworker stdout | 2025-02-07 03:32:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:32:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:32:40,835 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:32:40,891 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:32:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:32:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:11.831521+00:00 (in 29.999540 seconds)
autopruneworker stdout | 2025-02-07 03:32:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:11 UTC)" (scheduled at 2025-02-07 03:32:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:32:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895561836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:32:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:32:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:32:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:32:41,988 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:42,320 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:32:42,979 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:32:43,080 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:32:43,096 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:32:43,474 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:32:43,994 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:32:44,046 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:32:44,067 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:32:44,235 [236] [DEBUG] [app] Starting request: urn:request:fd42e083-4c54-4282-8b5d-e1b2f2081a51 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:44,235 [235] [DEBUG] [app] Starting request: urn:request:34169c58-9643-444b-9662-98fc23705afc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:32:44,238 [246] [DEBUG] [app] Starting request: urn:request:c50ed62e-8c90-4684-bd1a-522d9ba3f564 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:44,238 [239] [DEBUG] [app] Starting request: urn:request:94482717-fa3a-47e8-9087-8df4868ff5cf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:44,238 [246] [DEBUG] [app] Ending request: urn:request:c50ed62e-8c90-4684-bd1a-522d9ba3f564 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c50ed62e-8c90-4684-bd1a-522d9ba3f564', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:32:44,239 [239] [DEBUG] [app] Ending request: urn:request:94482717-fa3a-47e8-9087-8df4868ff5cf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:94482717-fa3a-47e8-9087-8df4868ff5cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:32:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:32:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:32:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:32:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:44,241 [235] [DEBUG] [app] Starting request: urn:request:4fdf336f-6f03-4b52-9a27-5799b9dde0d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:44,242 [235] [DEBUG] [app] Ending request: urn:request:4fdf336f-6f03-4b52-9a27-5799b9dde0d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4fdf336f-6f03-4b52-9a27-5799b9dde0d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:32:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:32:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:44,242 [238] [DEBUG] [app] Starting request: urn:request:da61c2cc-a6b5-46cf-a53e-1b95bb657a1d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:44,243 [238] [DEBUG] [app] Ending request: urn:request:da61c2cc-a6b5-46cf-a53e-1b95bb657a1d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:da61c2cc-a6b5-46cf-a53e-1b95bb657a1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:32:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:32:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:44,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:44,254 [235] [DEBUG] [app] Ending request: urn:request:34169c58-9643-444b-9662-98fc23705afc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:34169c58-9643-444b-9662-98fc23705afc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:32:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:32:44,255 [236] [DEBUG] [app] Ending request: urn:request:fd42e083-4c54-4282-8b5d-e1b2f2081a51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd42e083-4c54-4282-8b5d-e1b2f2081a51', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:32:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
namespacegcworker stdout | 2025-02-07 03:32:45,737 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:32:45,816 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:32:45,819 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:32:45,823 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:32:45,826 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:32:45,829 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:32:46,155 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:32:46,366 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:32:46,846 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:32:46,849 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:32:46,851 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:32:47,787 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:32:47,790 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:32:47,801 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:32:47,806 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:32:47,810 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:32:47,813 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:32:47,815 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:32:47,820 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:32:47,824 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:32:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:50.814677+00:00 (in 2.082033 seconds)
notificationworker stdout | 2025-02-07 03:32:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:58 UTC)" (scheduled at 2025-02-07 03:32:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:32:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:32:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 48, 732929), True, datetime.datetime(2025, 2, 7, 3, 32, 48, 732929), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:32:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:32:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:32:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:32:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:32:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:32:58.732147+00:00 (in 7.916998 seconds)
notificationworker stdout | 2025-02-07 03:32:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:50 UTC)" (scheduled at 2025-02-07 03:32:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:32:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:32:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:32:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:32:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:10.424250+00:00 (in 16.000717 seconds)
gcworker stdout | 2025-02-07 03:32:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:24 UTC)" (scheduled at 2025-02-07 03:32:54.423055+00:00)
gcworker stdout | 2025-02-07 03:32:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:32:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:32:55,336 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:32:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:32:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:56.118855+00:00 (in 59.999485 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:32:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:56 UTC)" (scheduled at 2025-02-07 03:32:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:32:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:32:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:32:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:37.715878+00:00 (in 41.001893 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:56 UTC)" (scheduled at 2025-02-07 03:32:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:32:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:32:57,304 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:32:58,280 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:32:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:32:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:08.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 03:32:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:08 UTC)" (scheduled at 2025-02-07 03:32:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:32:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:32:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 32, 58, 732882), True, datetime.datetime(2025, 2, 7, 3, 32, 58, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:32:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:32:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:32:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:32:59,235 [235] [DEBUG] [app] Starting request: urn:request:c4f67f5f-2977-4968-b3d0-f01fbf2cc8d7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:59,235 [238] [DEBUG] [app] Starting request: urn:request:113bab6c-8f3a-4971-8a2d-d1c3b0d34a27 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:32:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:32:59,238 [239] [DEBUG] [app] Starting request: urn:request:aa55ed72-c397-459d-adaa-b4f631181ee9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:59,238 [239] [DEBUG] [app] Ending request: urn:request:aa55ed72-c397-459d-adaa-b4f631181ee9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa55ed72-c397-459d-adaa-b4f631181ee9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:32:59,238 [245] [DEBUG] [app] Starting request: urn:request:730489be-caed-4317-903d-dc15afd78637 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:32:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:32:59,239 [245] [DEBUG] [app] Ending request: urn:request:730489be-caed-4317-903d-dc15afd78637 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:730489be-caed-4317-903d-dc15afd78637', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 03:32:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:32:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:32:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:32:59,241 [237] [DEBUG] [app] Starting request: urn:request:b3e90efe-459a-4f15-ae70-856d0fe14a70 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:59,241 [237] [DEBUG] [app] Ending request: urn:request:b3e90efe-459a-4f15-ae70-856d0fe14a70 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3e90efe-459a-4f15-ae70-856d0fe14a70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:32:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:32:59,243 [235] [DEBUG] [app] Starting request: urn:request:5911fdb3-3526-4cf2-93c4-11155a0f8843 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:32:59,243 [235] [DEBUG] [app] Ending request: urn:request:5911fdb3-3526-4cf2-93c4-11155a0f8843 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5911fdb3-3526-4cf2-93c4-11155a0f8843', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:32:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:32:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:32:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:32:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:32:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:32:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:32:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:32:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:32:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:32:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:32:59,255 [235] [DEBUG] [app] Ending request: urn:request:c4f67f5f-2977-4968-b3d0-f01fbf2cc8d7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4f67f5f-2977-4968-b3d0-f01fbf2cc8d7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:32:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:32:59,256 [238] [DEBUG] [app] Ending request: urn:request:113bab6c-8f3a-4971-8a2d-d1c3b0d34a27 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:113bab6c-8f3a-4971-8a2d-d1c3b0d34a27', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:32:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:32:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:32:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:32:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:32:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:32:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:04.936688+00:00 (in 4.998331 seconds)
exportactionlogsworker stdout | 2025-02-07 03:32:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:59 UTC)" (scheduled at 2025-02-07 03:32:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:32:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:32:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:33:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:33:02,641 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:33:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:33:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:23.420886+00:00 (in 20.001273 seconds)
repositorygcworker stdout | 2025-02-07 03:33:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:03 UTC)" (scheduled at 2025-02-07 03:33:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:33:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:33:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 3, 419933), True, datetime.datetime(2025, 2, 7, 3, 33, 3, 419933), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:33:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:33:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:33:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:33:03,626 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:33:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:33:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:33.919286+00:00 (in 29.999567 seconds)
buildlogsarchiver stdout | 2025-02-07 03:33:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:33 UTC)" (scheduled at 2025-02-07 03:33:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:33:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 33, 3, 919975), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:33:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:33:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:33:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:33:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:33:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:59.937899+00:00 (in 55.000755 seconds)
exportactionlogsworker stdout | 2025-02-07 03:33:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:04 UTC)" (scheduled at 2025-02-07 03:33:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:33:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:33:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 4, 937448), True, datetime.datetime(2025, 2, 7, 3, 33, 4, 937448), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:33:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:33:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:33:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:33:08,284 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:33:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:33:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:18.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 03:33:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:18 UTC)" (scheduled at 2025-02-07 03:33:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:33:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:33:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 8, 732896), True, datetime.datetime(2025, 2, 7, 3, 33, 8, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:33:08,747 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:33:08,747 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:33:08,747 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:33:08,824 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:33:10,012 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:33:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:33:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:24.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 03:33:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:40 UTC)" (scheduled at 2025-02-07 03:33:10.424250+00:00)
gcworker stdout | 2025-02-07 03:33:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:33:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898890431, None, 1, 0])
gcworker stdout | 2025-02-07 03:33:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:33:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:33:10,843 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:33:10,901 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:33:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:33:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 03:33:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:41 UTC)" (scheduled at 2025-02-07 03:33:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:33:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895591836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:33:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:33:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:33:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:33:11,995 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:12,334 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:33:13,015 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:33:13,117 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:33:13,132 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:33:13,511 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:33:14,030 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:33:14,056 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:33:14,078 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:33:14,235 [236] [DEBUG] [app] Starting request: urn:request:586837a6-b0a6-474c-95c3-717ced6624bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:14,235 [235] [DEBUG] [app] Starting request: urn:request:8a0b3b00-f949-49c5-a945-14e232a11cad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:33:14,238 [246] [DEBUG] [app] Starting request: urn:request:131ce8f4-9918-48d2-8fd7-269364f9f550 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:14,238 [239] [DEBUG] [app] Starting request: urn:request:3bd926f4-3db2-4044-9dd4-5de748ca2e16 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:14,238 [239] [DEBUG] [app] Ending request: urn:request:3bd926f4-3db2-4044-9dd4-5de748ca2e16 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3bd926f4-3db2-4044-9dd4-5de748ca2e16', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:33:14,238 [246] [DEBUG] [app] Ending request: urn:request:131ce8f4-9918-48d2-8fd7-269364f9f550 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:131ce8f4-9918-48d2-8fd7-269364f9f550', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:33:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:33:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:14,241 [236] [DEBUG] [app] Starting request: urn:request:74a0763d-d19c-40eb-a5b5-9ffa9092d8bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:33:14,241 [237] [DEBUG] [app] Starting request: urn:request:41cb3391-1609-4b75-a8d1-d726114e422d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:33:14,241 [236] [DEBUG] [app] Ending request: urn:request:74a0763d-d19c-40eb-a5b5-9ffa9092d8bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:74a0763d-d19c-40eb-a5b5-9ffa9092d8bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:33:14,242 [237] [DEBUG] [app] Ending request: urn:request:41cb3391-1609-4b75-a8d1-d726114e422d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:41cb3391-1609-4b75-a8d1-d726114e422d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:33:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:33:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:14,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:14,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:14,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:14,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:14,253 [235] [DEBUG] [app] Ending request: urn:request:8a0b3b00-f949-49c5-a945-14e232a11cad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a0b3b00-f949-49c5-a945-14e232a11cad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:33:14,253 [236] [DEBUG] [app] Ending request: urn:request:586837a6-b0a6-474c-95c3-717ced6624bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:586837a6-b0a6-474c-95c3-717ced6624bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:33:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:33:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:33:15,748 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:33:15,824 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:33:15,827 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:33:15,830 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:33:15,833 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:33:15,836 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:33:16,191 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:33:16,402 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:33:16,854 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:33:16,857 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:33:16,861 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:33:17,798 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:33:17,801 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:33:17,808 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:33:17,818 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:33:17,822 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:33:17,825 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:33:17,829 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:33:17,832 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:33:17,834 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:33:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:33:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:28.732147+00:00 (in 9.999541 seconds)
notificationworker stdout | 2025-02-07 03:33:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:28 UTC)" (scheduled at 2025-02-07 03:33:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:33:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:33:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 18, 732885), True, datetime.datetime(2025, 2, 7, 3, 33, 18, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:33:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:33:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:33:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:33:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:33:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:32.420707+00:00 (in 8.999279 seconds)
repositorygcworker stdout | 2025-02-07 03:33:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:23 UTC)" (scheduled at 2025-02-07 03:33:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:33:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:33:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:33:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:33:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:40.424250+00:00 (in 16.000741 seconds)
gcworker stdout | 2025-02-07 03:33:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:54 UTC)" (scheduled at 2025-02-07 03:33:24.423055+00:00)
gcworker stdout | 2025-02-07 03:33:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:33:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:33:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:33:25,373 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:33:27,340 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:33:28,314 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:33:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:33:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:38.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 03:33:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:38 UTC)" (scheduled at 2025-02-07 03:33:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:33:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:33:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 28, 732855), True, datetime.datetime(2025, 2, 7, 3, 33, 28, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:33:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:33:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:33:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:33:29,235 [238] [DEBUG] [app] Starting request: urn:request:893e4ece-7e50-4056-aae9-b93e2b3109bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:29,235 [235] [DEBUG] [app] Starting request: urn:request:1154c5c0-b8d1-4ed2-a22b-abfbd380df9b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:33:29,238 [239] [DEBUG] [app] Starting request: urn:request:11428c26-5893-4a3e-80c1-45e538f5cebf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:29,239 [246] [DEBUG] [app] Starting request: urn:request:bcf1b8e4-1f77-49cd-81d9-c135be8b42f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:29,239 [239] [DEBUG] [app] Ending request: urn:request:11428c26-5893-4a3e-80c1-45e538f5cebf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11428c26-5893-4a3e-80c1-45e538f5cebf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:33:29,239 [246] [DEBUG] [app] Ending request: urn:request:bcf1b8e4-1f77-49cd-81d9-c135be8b42f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bcf1b8e4-1f77-49cd-81d9-c135be8b42f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:33:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:33:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:33:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:29,242 [237] [DEBUG] [app] Starting request: urn:request:18b5c24c-794a-481a-add9-6e86022722d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:33:29,242 [237] [DEBUG] [app] Ending request: urn:request:18b5c24c-794a-481a-add9-6e86022722d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18b5c24c-794a-481a-add9-6e86022722d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:33:29,243 [238] [DEBUG] [app] Starting request: urn:request:db610fc2-2c05-477f-92a5-313731f9b055 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:33:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:29,243 [238] [DEBUG] [app] Ending request: urn:request:db610fc2-2c05-477f-92a5-313731f9b055 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db610fc2-2c05-477f-92a5-313731f9b055', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:33:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:33:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:29,256 [235] [DEBUG] [app] Ending request: urn:request:1154c5c0-b8d1-4ed2-a22b-abfbd380df9b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1154c5c0-b8d1-4ed2-a22b-abfbd380df9b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:29,256 [238] [DEBUG] [app] Ending request: urn:request:893e4ece-7e50-4056-aae9-b93e2b3109bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:893e4ece-7e50-4056-aae9-b93e2b3109bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:33:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:33:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:33:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:33:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:33:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:32.625945+00:00 (in 1.004332 seconds)
namespacegcworker stdout | 2025-02-07 03:33:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:31 UTC)" (scheduled at 2025-02-07 03:33:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:33:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:33:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 31, 621889), True, datetime.datetime(2025, 2, 7, 3, 33, 31, 621889), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:33:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:33:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:33:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:33:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:33:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:03.419110+00:00 (in 30.997960 seconds)
repositorygcworker stdout | 2025-02-07 03:33:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:38:32 UTC)" (scheduled at 2025-02-07 03:33:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:33:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 33, 32, 421439), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:33:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 33, 32, 421439), True, datetime.datetime(2025, 2, 7, 3, 33, 32, 421439), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:33:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 33, 32, 421439), True, datetime.datetime(2025, 2, 7, 3, 33, 32, 421439), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 33, 32, 421439), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:33:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:33:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:38:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:33:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:33:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:31.621131+00:00 (in 58.994742 seconds)
namespacegcworker stdout | 2025-02-07 03:33:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:32 UTC)" (scheduled at 2025-02-07 03:33:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:33:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:33:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:33:32,678 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:33:33,646 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:33:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:33:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:03.919286+00:00 (in 29.999512 seconds)
buildlogsarchiver stdout | 2025-02-07 03:33:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:03 UTC)" (scheduled at 2025-02-07 03:33:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:33:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 33, 33, 920030), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:33:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:33:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:33:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:56.713547+00:00 (in 18.997220 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:37 UTC)" (scheduled at 2025-02-07 03:33:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:33:38,320 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:33:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:33:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:48.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:33:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:48 UTC)" (scheduled at 2025-02-07 03:33:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:33:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:33:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 38, 732862), True, datetime.datetime(2025, 2, 7, 3, 33, 38, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:33:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:33:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:33:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:33:38,853 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:33:40,025 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:33:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:33:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:54.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 03:33:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:10 UTC)" (scheduled at 2025-02-07 03:33:40.424250+00:00)
gcworker stdout | 2025-02-07 03:33:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:33:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898920431, None, 1, 0])
gcworker stdout | 2025-02-07 03:33:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:33:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:33:40,879 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:33:40,934 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:33:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:33:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:11.831521+00:00 (in 29.999552 seconds)
autopruneworker stdout | 2025-02-07 03:33:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:11 UTC)" (scheduled at 2025-02-07 03:33:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:33:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895621836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:33:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:33:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:33:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:33:42,006 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:42,370 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:33:43,052 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:33:43,140 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:33:43,169 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:33:43,547 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
manifestbackfillworker stdout | 2025-02-07 03:33:44,068 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gcworker stdout | 2025-02-07 03:33:44,071 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:33:44,107 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:33:44,235 [235] [DEBUG] [app] Starting request: urn:request:faec55d4-2a1d-4151-920f-2ddfe0073e5c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:44,235 [238] [DEBUG] [app] Starting request: urn:request:fea7bac4-1e7d-4831-959c-c3b660be8880 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:33:44,239 [239] [DEBUG] [app] Starting request: urn:request:c37a343c-c575-4c48-aecf-c308c1d78911 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:44,239 [243] [DEBUG] [app] Starting request: urn:request:edeedc97-e832-4401-8a7b-d422bc22c369 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:44,239 [239] [DEBUG] [app] Ending request: urn:request:c37a343c-c575-4c48-aecf-c308c1d78911 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c37a343c-c575-4c48-aecf-c308c1d78911', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:33:44,239 [243] [DEBUG] [app] Ending request: urn:request:edeedc97-e832-4401-8a7b-d422bc22c369 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:edeedc97-e832-4401-8a7b-d422bc22c369', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:33:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:33:44,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:44,242 [238] [DEBUG] [app] Starting request: urn:request:59ea7926-b840-484d-840a-9f49264534cd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:33:44,242 [238] [DEBUG] [app] Ending request: urn:request:59ea7926-b840-484d-840a-9f49264534cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59ea7926-b840-484d-840a-9f49264534cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:33:44,242 [236] [DEBUG] [app] Starting request: urn:request:59c93db9-6b87-47c4-8616-18cc9f2f6bea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:33:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:44,242 [236] [DEBUG] [app] Ending request: urn:request:59c93db9-6b87-47c4-8616-18cc9f2f6bea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59c93db9-6b87-47c4-8616-18cc9f2f6bea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:33:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:33:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:44,254 [235] [DEBUG] [app] Ending request: urn:request:faec55d4-2a1d-4151-920f-2ddfe0073e5c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:faec55d4-2a1d-4151-920f-2ddfe0073e5c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:44,254 [238] [DEBUG] [app] Ending request: urn:request:fea7bac4-1e7d-4831-959c-c3b660be8880 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fea7bac4-1e7d-4831-959c-c3b660be8880', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:33:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:33:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:33:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:33:45,774 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:33:45,833 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:33:45,836 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:33:45,839 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:33:45,842 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:33:45,844 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:33:46,227 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:33:46,418 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:33:46,863 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:33:46,866 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:33:46,869 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:33:47,810 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:33:47,812 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:33:47,815 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:33:47,825 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:33:47,832 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:33:47,835 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:33:47,839 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:33:47,842 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:33:47,844 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:33:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:33:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:50.814677+00:00 (in 2.082066 seconds)
notificationworker stdout | 2025-02-07 03:33:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:58 UTC)" (scheduled at 2025-02-07 03:33:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:33:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:33:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 48, 732902), True, datetime.datetime(2025, 2, 7, 3, 33, 48, 732902), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:33:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:33:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:33:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:33:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:33:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:33:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:33:58.732147+00:00 (in 7.916993 seconds)
notificationworker stdout | 2025-02-07 03:33:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:50 UTC)" (scheduled at 2025-02-07 03:33:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:33:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:33:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:33:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:33:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:10.424250+00:00 (in 16.000670 seconds)
gcworker stdout | 2025-02-07 03:33:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:24 UTC)" (scheduled at 2025-02-07 03:33:54.423055+00:00)
gcworker stdout | 2025-02-07 03:33:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:33:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:33:55,386 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:33:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:33:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:56.118855+00:00 (in 59.999486 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:33:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:56 UTC)" (scheduled at 2025-02-07 03:33:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:33:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:33:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:33:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:37.715878+00:00 (in 41.001875 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:56 UTC)" (scheduled at 2025-02-07 03:33:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:33:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:33:57,353 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:33:58,331 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:33:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:33:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:08.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 03:33:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:08 UTC)" (scheduled at 2025-02-07 03:33:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:33:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:33:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 33, 58, 732893), True, datetime.datetime(2025, 2, 7, 3, 33, 58, 732893), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:33:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:33:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:33:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:33:59,235 [237] [DEBUG] [app] Starting request: urn:request:1fdc0e62-3d21-4594-83b6-9fe4f10952bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:59,235 [235] [DEBUG] [app] Starting request: urn:request:0f4208b0-9e24-47db-bf86-09ac8e083eca (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:33:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:33:59,238 [246] [DEBUG] [app] Starting request: urn:request:ebba2097-1d7f-42ce-9580-1d2c8a93bce2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:59,238 [239] [DEBUG] [app] Starting request: urn:request:6b4edda4-5a64-4189-8205-18be4fd1c89b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:33:59,239 [246] [DEBUG] [app] Ending request: urn:request:ebba2097-1d7f-42ce-9580-1d2c8a93bce2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ebba2097-1d7f-42ce-9580-1d2c8a93bce2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:33:59,239 [239] [DEBUG] [app] Ending request: urn:request:6b4edda4-5a64-4189-8205-18be4fd1c89b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b4edda4-5a64-4189-8205-18be4fd1c89b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:33:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:33:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:33:59,242 [236] [DEBUG] [app] Starting request: urn:request:da38d2c7-0823-4a0b-b1c9-342ae9ae35ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:33:59,242 [238] [DEBUG] [app] Starting request: urn:request:e6fc422b-990e-4311-9427-a7915196b293 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:33:59,242 [238] [DEBUG] [app] Ending request: urn:request:e6fc422b-990e-4311-9427-a7915196b293 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6fc422b-990e-4311-9427-a7915196b293', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:33:59,242 [236] [DEBUG] [app] Ending request: urn:request:da38d2c7-0823-4a0b-b1c9-342ae9ae35ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:da38d2c7-0823-4a0b-b1c9-342ae9ae35ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:33:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:33:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:33:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:33:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:33:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:33:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:33:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:33:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:33:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:33:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:33:59,254 [235] [DEBUG] [app] Ending request: urn:request:0f4208b0-9e24-47db-bf86-09ac8e083eca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f4208b0-9e24-47db-bf86-09ac8e083eca', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:33:59,254 [237] [DEBUG] [app] Ending request: urn:request:1fdc0e62-3d21-4594-83b6-9fe4f10952bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1fdc0e62-3d21-4594-83b6-9fe4f10952bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:33:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:33:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:33:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:33:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:33:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 03:33:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:33:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:04.936688+00:00 (in 4.998287 seconds)
exportactionlogsworker stdout | 2025-02-07 03:33:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:59 UTC)" (scheduled at 2025-02-07 03:33:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:33:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:33:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:34:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:34:02,706 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:34:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:34:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:23.420886+00:00 (in 20.001300 seconds)
repositorygcworker stdout | 2025-02-07 03:34:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:03 UTC)" (scheduled at 2025-02-07 03:34:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:34:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:34:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 3, 419878), True, datetime.datetime(2025, 2, 7, 3, 34, 3, 419878), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:34:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:34:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:34:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:34:03,665 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:34:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:34:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:33.919286+00:00 (in 29.999540 seconds)
buildlogsarchiver stdout | 2025-02-07 03:34:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:33 UTC)" (scheduled at 2025-02-07 03:34:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:34:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 34, 3, 920028), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:34:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:34:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:34:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:34:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:34:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:52.937648+00:00 (in 48.000521 seconds)
exportactionlogsworker stdout | 2025-02-07 03:34:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:04 UTC)" (scheduled at 2025-02-07 03:34:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:34:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:34:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 4, 937483), True, datetime.datetime(2025, 2, 7, 3, 34, 4, 937483), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:34:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:34:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:34:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:34:08,343 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:34:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:34:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:18.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 03:34:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:18 UTC)" (scheduled at 2025-02-07 03:34:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:34:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:34:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 8, 732873), True, datetime.datetime(2025, 2, 7, 3, 34, 8, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:34:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:34:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:34:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:34:08,889 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:34:10,032 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:34:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:34:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:24.423055+00:00 (in 13.998325 seconds)
gcworker stdout | 2025-02-07 03:34:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:40 UTC)" (scheduled at 2025-02-07 03:34:10.424250+00:00)
gcworker stdout | 2025-02-07 03:34:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:34:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898950431, None, 1, 0])
gcworker stdout | 2025-02-07 03:34:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:34:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:34:10,908 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:34:10,960 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:34:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:34:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:41.831521+00:00 (in 29.999508 seconds)
autopruneworker stdout | 2025-02-07 03:34:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:41 UTC)" (scheduled at 2025-02-07 03:34:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:34:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895651836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:34:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:34:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:34:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:34:12,042 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:12,407 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:34:13,088 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:34:13,176 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:34:13,205 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:34:13,584 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:34:14,078 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:34:14,103 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:34:14,142 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:34:14,235 [236] [DEBUG] [app] Starting request: urn:request:44d9e936-dbc9-4aea-bbfe-e3cb3e2e0b7a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:14,235 [237] [DEBUG] [app] Starting request: urn:request:f4c60e55-4ee8-421b-af95-a24145254d7b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:34:14,238 [239] [DEBUG] [app] Starting request: urn:request:7356fc85-76d3-4c9d-a725-e3c2d353b69f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:14,239 [245] [DEBUG] [app] Starting request: urn:request:02f480b4-693b-435f-b69b-4d508cf2114e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:14,239 [239] [DEBUG] [app] Ending request: urn:request:7356fc85-76d3-4c9d-a725-e3c2d353b69f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7356fc85-76d3-4c9d-a725-e3c2d353b69f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:34:14,239 [245] [DEBUG] [app] Ending request: urn:request:02f480b4-693b-435f-b69b-4d508cf2114e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02f480b4-693b-435f-b69b-4d508cf2114e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:34:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:34:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:34:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:14,242 [238] [DEBUG] [app] Starting request: urn:request:ee0bc95b-1a74-4c62-8716-387189c09326 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:34:14,242 [238] [DEBUG] [app] Ending request: urn:request:ee0bc95b-1a74-4c62-8716-387189c09326 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee0bc95b-1a74-4c62-8716-387189c09326', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:34:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:34:14,243 [235] [DEBUG] [app] Starting request: urn:request:c605a4d2-e959-48e5-a1eb-90f601049c22 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:34:14,243 [235] [DEBUG] [app] Ending request: urn:request:c605a4d2-e959-48e5-a1eb-90f601049c22 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c605a4d2-e959-48e5-a1eb-90f601049c22', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:34:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:34:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:14,254 [236] [DEBUG] [app] Ending request: urn:request:44d9e936-dbc9-4aea-bbfe-e3cb3e2e0b7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44d9e936-dbc9-4aea-bbfe-e3cb3e2e0b7a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:14,254 [237] [DEBUG] [app] Ending request: urn:request:f4c60e55-4ee8-421b-af95-a24145254d7b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f4c60e55-4ee8-421b-af95-a24145254d7b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:34:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:34:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:34:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:34:15,810 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:34:15,841 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:34:15,845 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:34:15,847 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:34:15,850 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:34:15,853 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:34:16,258 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:34:16,454 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:34:16,871 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:34:16,874 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:34:16,878 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:34:17,822 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:34:17,825 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:34:17,828 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:34:17,834 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:34:17,843 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:34:17,845 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:34:17,848 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:34:17,851 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:34:17,854 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:34:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:34:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:28.732147+00:00 (in 9.999531 seconds)
notificationworker stdout | 2025-02-07 03:34:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:28 UTC)" (scheduled at 2025-02-07 03:34:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:34:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:34:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 18, 732930), True, datetime.datetime(2025, 2, 7, 3, 34, 18, 732930), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:34:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:34:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:34:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:34:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:34:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:03.419110+00:00 (in 39.997740 seconds)
repositorygcworker stdout | 2025-02-07 03:34:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:23 UTC)" (scheduled at 2025-02-07 03:34:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:34:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:34:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:34:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:34:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:40.424250+00:00 (in 16.000706 seconds)
gcworker stdout | 2025-02-07 03:34:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:54 UTC)" (scheduled at 2025-02-07 03:34:24.423055+00:00)
gcworker stdout | 2025-02-07 03:34:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:34:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:34:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:34:25,423 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:34:27,380 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:34:28,368 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:34:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:34:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:38.732147+00:00 (in 9.999471 seconds)
notificationworker stdout | 2025-02-07 03:34:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:38 UTC)" (scheduled at 2025-02-07 03:34:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:34:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:34:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 28, 733003), True, datetime.datetime(2025, 2, 7, 3, 34, 28, 733003), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:34:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:34:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:34:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:34:29,236 [236] [DEBUG] [app] Starting request: urn:request:7add63f5-488e-42b1-8dfb-d327581cba7c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:29,236 [238] [DEBUG] [app] Starting request: urn:request:12049257-8bfe-4988-aba4-e5a0942be9a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:34:29,239 [239] [DEBUG] [app] Starting request: urn:request:befe0be9-9cb7-42cc-9b28-ab1f5710ac5b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:29,239 [245] [DEBUG] [app] Starting request: urn:request:d5891c0f-ee6b-45ed-bde6-0d5769d8ab4b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:29,240 [245] [DEBUG] [app] Ending request: urn:request:d5891c0f-ee6b-45ed-bde6-0d5769d8ab4b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d5891c0f-ee6b-45ed-bde6-0d5769d8ab4b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:34:29,240 [239] [DEBUG] [app] Ending request: urn:request:befe0be9-9cb7-42cc-9b28-ab1f5710ac5b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:befe0be9-9cb7-42cc-9b28-ab1f5710ac5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:34:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:34:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:29,243 [236] [DEBUG] [app] Starting request: urn:request:b463c357-d77a-4ebd-a0f0-ea6269ebff93 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:34:29,243 [235] [DEBUG] [app] Starting request: urn:request:91ffe79a-c9d4-44c1-ab15-7849500addcd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:34:29,243 [235] [DEBUG] [app] Ending request: urn:request:91ffe79a-c9d4-44c1-ab15-7849500addcd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91ffe79a-c9d4-44c1-ab15-7849500addcd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:34:29,243 [236] [DEBUG] [app] Ending request: urn:request:b463c357-d77a-4ebd-a0f0-ea6269ebff93 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b463c357-d77a-4ebd-a0f0-ea6269ebff93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:34:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:34:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:29,255 [236] [DEBUG] [app] Ending request: urn:request:7add63f5-488e-42b1-8dfb-d327581cba7c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7add63f5-488e-42b1-8dfb-d327581cba7c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:34:29,255 [238] [DEBUG] [app] Ending request: urn:request:12049257-8bfe-4988-aba4-e5a0942be9a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:12049257-8bfe-4988-aba4-e5a0942be9a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:34:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:34:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:34:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:34:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:32.625945+00:00 (in 1.004351 seconds)
namespacegcworker stdout | 2025-02-07 03:34:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:31 UTC)" (scheduled at 2025-02-07 03:34:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:34:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:34:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 31, 621905), True, datetime.datetime(2025, 2, 7, 3, 34, 31, 621905), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:34:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:34:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:34:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:34:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:34:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:31.621131+00:00 (in 58.994749 seconds)
namespacegcworker stdout | 2025-02-07 03:34:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:32 UTC)" (scheduled at 2025-02-07 03:34:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:34:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:34:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:34:32,742 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:34:33,702 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:34:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:34:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:03.919286+00:00 (in 29.999512 seconds)
buildlogsarchiver stdout | 2025-02-07 03:34:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:03 UTC)" (scheduled at 2025-02-07 03:34:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:34:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 34, 33, 920022), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:34:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:34:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:34:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:56.713547+00:00 (in 18.997209 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:37 UTC)" (scheduled at 2025-02-07 03:34:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:34:38,378 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:34:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:34:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:48.732147+00:00 (in 9.999500 seconds)
notificationworker stdout | 2025-02-07 03:34:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:48 UTC)" (scheduled at 2025-02-07 03:34:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:34:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:34:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 38, 732952), True, datetime.datetime(2025, 2, 7, 3, 34, 38, 732952), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:34:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:34:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:34:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:34:38,925 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:34:40,069 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:34:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:34:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:54.423055+00:00 (in 13.998326 seconds)
gcworker stdout | 2025-02-07 03:34:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:10 UTC)" (scheduled at 2025-02-07 03:34:40.424250+00:00)
gcworker stdout | 2025-02-07 03:34:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:34:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738898980431, None, 1, 0])
gcworker stdout | 2025-02-07 03:34:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:34:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:34:40,945 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:34:40,997 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:34:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:34:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:11.831521+00:00 (in 29.999538 seconds)
autopruneworker stdout | 2025-02-07 03:34:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:11 UTC)" (scheduled at 2025-02-07 03:34:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:34:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895681836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:34:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:34:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:34:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:34:42,079 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:42,426 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:34:43,124 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:34:43,211 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:34:43,241 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:34:43,618 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:34:44,090 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:34:44,139 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:34:44,179 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:34:44,235 [237] [DEBUG] [app] Starting request: urn:request:14be417e-43a3-4a66-9e4d-e494344d9e95 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:44,235 [235] [DEBUG] [app] Starting request: urn:request:a467a9c7-8544-4439-b177-f5d6881a372a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:34:44,239 [245] [DEBUG] [app] Starting request: urn:request:f714c9ae-b946-4783-b776-c79624394e4f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:44,239 [239] [DEBUG] [app] Starting request: urn:request:54593753-4e4d-4861-aad0-31857d5cd75b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:44,239 [245] [DEBUG] [app] Ending request: urn:request:f714c9ae-b946-4783-b776-c79624394e4f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f714c9ae-b946-4783-b776-c79624394e4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:34:44,239 [239] [DEBUG] [app] Ending request: urn:request:54593753-4e4d-4861-aad0-31857d5cd75b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54593753-4e4d-4861-aad0-31857d5cd75b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:34:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:34:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:34:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:44,243 [235] [DEBUG] [app] Starting request: urn:request:ba74720e-2bac-46be-8af8-e72a7ff64138 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:34:44,243 [235] [DEBUG] [app] Ending request: urn:request:ba74720e-2bac-46be-8af8-e72a7ff64138 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba74720e-2bac-46be-8af8-e72a7ff64138', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:34:44,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:44,245 [236] [DEBUG] [app] Starting request: urn:request:e9a600c5-a327-498e-a917-b9315a6a924c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:34:44,246 [236] [DEBUG] [app] Ending request: urn:request:e9a600c5-a327-498e-a917-b9315a6a924c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e9a600c5-a327-498e-a917-b9315a6a924c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:34:44,246 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:44,246 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:44,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:44,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:44,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:44,251 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:44,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:44,256 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:44,256 [237] [DEBUG] [app] Ending request: urn:request:14be417e-43a3-4a66-9e4d-e494344d9e95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14be417e-43a3-4a66-9e4d-e494344d9e95', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:34:44,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:34:44,257 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:44,258 [235] [DEBUG] [app] Ending request: urn:request:a467a9c7-8544-4439-b177-f5d6881a372a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a467a9c7-8544-4439-b177-f5d6881a372a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:44,259 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gunicorn-web stdout | 2025-02-07 03:34:44,259 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:34:45,847 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-web stdout | 2025-02-07 03:34:45,851 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:34:45,856 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:34:45,859 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:34:45,862 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:34:45,864 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
buildlogsarchiver stdout | 2025-02-07 03:34:46,265 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:34:46,473 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:34:46,881 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:34:46,886 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:34:46,889 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:34:47,833 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:34:47,836 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:34:47,838 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:34:47,843 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:34:47,851 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:34:47,855 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:34:47,858 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:34:47,861 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:34:47,864 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:34:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:34:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:50.814677+00:00 (in 2.082060 seconds)
notificationworker stdout | 2025-02-07 03:34:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:58 UTC)" (scheduled at 2025-02-07 03:34:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:34:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:34:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 48, 732896), True, datetime.datetime(2025, 2, 7, 3, 34, 48, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:34:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:34:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:34:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:34:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:34:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:34:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:58.732147+00:00 (in 7.917015 seconds)
notificationworker stdout | 2025-02-07 03:34:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:50 UTC)" (scheduled at 2025-02-07 03:34:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:34:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:34:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:34:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:34:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:34:59.937899+00:00 (in 6.999805 seconds)
exportactionlogsworker stdout | 2025-02-07 03:34:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:39:52 UTC)" (scheduled at 2025-02-07 03:34:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:34:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 34, 52, 938371), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:34:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 34, 52, 938371), True, datetime.datetime(2025, 2, 7, 3, 34, 52, 938371), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:34:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 34, 52, 938371), True, datetime.datetime(2025, 2, 7, 3, 34, 52, 938371), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 34, 52, 938371), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:34:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:34:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:39:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:34:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:34:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:10.424250+00:00 (in 16.000720 seconds)
gcworker stdout | 2025-02-07 03:34:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:24 UTC)" (scheduled at 2025-02-07 03:34:54.423055+00:00)
gcworker stdout | 2025-02-07 03:34:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:34:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:34:55,449 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:34:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:34:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:56.118855+00:00 (in 59.999480 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:34:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:56 UTC)" (scheduled at 2025-02-07 03:34:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:34:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:34:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:34:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:37.715878+00:00 (in 41.001867 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:56 UTC)" (scheduled at 2025-02-07 03:34:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:34:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:34:57,402 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:34:58,392 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:34:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:34:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:08.732147+00:00 (in 9.999502 seconds)
notificationworker stdout | 2025-02-07 03:34:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:08 UTC)" (scheduled at 2025-02-07 03:34:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:34:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:34:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 34, 58, 732919), True, datetime.datetime(2025, 2, 7, 3, 34, 58, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:34:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:34:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:34:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:34:59,235 [237] [DEBUG] [app] Starting request: urn:request:5a1a77c6-92b9-41fe-bf87-3cbfdf20f991 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:59,235 [238] [DEBUG] [app] Starting request: urn:request:543a9808-bbb3-4a29-89df-f1549a2bdfb3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:34:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:34:59,238 [239] [DEBUG] [app] Starting request: urn:request:771b265d-45c6-46e9-802c-6c43266859b6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:59,238 [246] [DEBUG] [app] Starting request: urn:request:c7882309-c825-4998-94c4-1d174c61d4c9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:34:59,239 [246] [DEBUG] [app] Ending request: urn:request:c7882309-c825-4998-94c4-1d174c61d4c9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c7882309-c825-4998-94c4-1d174c61d4c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:34:59,239 [239] [DEBUG] [app] Ending request: urn:request:771b265d-45c6-46e9-802c-6c43266859b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:771b265d-45c6-46e9-802c-6c43266859b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:34:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:34:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:34:59,243 [237] [DEBUG] [app] Starting request: urn:request:873e4c03-3051-423d-9f8f-468948ff60be (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:34:59,243 [237] [DEBUG] [app] Ending request: urn:request:873e4c03-3051-423d-9f8f-468948ff60be (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:873e4c03-3051-423d-9f8f-468948ff60be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:34:59,243 [235] [DEBUG] [app] Starting request: urn:request:058084ba-f510-439f-aacf-33f1acd34e92 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:34:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:34:59,244 [235] [DEBUG] [app] Ending request: urn:request:058084ba-f510-439f-aacf-33f1acd34e92 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:058084ba-f510-439f-aacf-33f1acd34e92', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:34:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:59,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:34:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:34:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:34:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:34:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:34:59,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:34:59,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:59,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:59,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:34:59,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:34:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:34:59,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:59,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:34:59,256 [237] [DEBUG] [app] Ending request: urn:request:5a1a77c6-92b9-41fe-bf87-3cbfdf20f991 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5a1a77c6-92b9-41fe-bf87-3cbfdf20f991', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:59,256 [238] [DEBUG] [app] Ending request: urn:request:543a9808-bbb3-4a29-89df-f1549a2bdfb3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:543a9808-bbb3-4a29-89df-f1549a2bdfb3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:34:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:34:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:34:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:34:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:34:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:34:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:34:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:34:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:04.936688+00:00 (in 4.998332 seconds)
exportactionlogsworker stdout | 2025-02-07 03:34:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:59 UTC)" (scheduled at 2025-02-07 03:34:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:34:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:34:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:35:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:35:02,758 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:35:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:35:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:23.420886+00:00 (in 20.001273 seconds)
repositorygcworker stdout | 2025-02-07 03:35:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:03 UTC)" (scheduled at 2025-02-07 03:35:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:35:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:35:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 3, 419906), True, datetime.datetime(2025, 2, 7, 3, 35, 3, 419906), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:35:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:35:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:35:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:35:03,731 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:35:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:35:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:33.919286+00:00 (in 29.999508 seconds)
buildlogsarchiver stdout | 2025-02-07 03:35:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:33 UTC)" (scheduled at 2025-02-07 03:35:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:35:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 35, 3, 920037), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:35:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:35:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:35:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:35:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:35:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:59.937899+00:00 (in 55.000752 seconds)
exportactionlogsworker stdout | 2025-02-07 03:35:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:04 UTC)" (scheduled at 2025-02-07 03:35:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:35:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:35:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 4, 937467), True, datetime.datetime(2025, 2, 7, 3, 35, 4, 937467), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:35:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:35:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:35:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:35:08,414 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:35:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:35:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:18.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 03:35:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:18 UTC)" (scheduled at 2025-02-07 03:35:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:35:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:35:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 8, 732855), True, datetime.datetime(2025, 2, 7, 3, 35, 8, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:35:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:35:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:35:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:35:08,932 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:35:10,105 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:35:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:35:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:24.423055+00:00 (in 13.998350 seconds)
gcworker stdout | 2025-02-07 03:35:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:40 UTC)" (scheduled at 2025-02-07 03:35:10.424250+00:00)
gcworker stdout | 2025-02-07 03:35:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:35:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899010431, None, 1, 0])
gcworker stdout | 2025-02-07 03:35:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:35:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:35:10,971 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:35:11,022 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:35:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:35:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:41.831521+00:00 (in 29.999545 seconds)
autopruneworker stdout | 2025-02-07 03:35:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:41 UTC)" (scheduled at 2025-02-07 03:35:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:35:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895711836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:35:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:35:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:35:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:35:12,092 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:12,450 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:35:13,157 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:35:13,234 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:35:13,273 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:35:13,655 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:35:14,126 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:35:14,175 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:35:14,215 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:35:14,236 [238] [DEBUG] [app] Starting request: urn:request:9dedb1b0-54aa-47b8-8032-f8c53032a6be (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:14,236 [235] [DEBUG] [app] Starting request: urn:request:b7da2bba-5415-4023-aa84-f34dae3b8c7c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:35:14,239 [239] [DEBUG] [app] Starting request: urn:request:a1998351-eeef-42ae-9de4-44535e7fddf5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:14,239 [245] [DEBUG] [app] Starting request: urn:request:ba283e72-aee4-4019-99f6-9661786a60be (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:14,239 [245] [DEBUG] [app] Ending request: urn:request:ba283e72-aee4-4019-99f6-9661786a60be (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ba283e72-aee4-4019-99f6-9661786a60be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:35:14,239 [239] [DEBUG] [app] Ending request: urn:request:a1998351-eeef-42ae-9de4-44535e7fddf5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a1998351-eeef-42ae-9de4-44535e7fddf5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:35:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:35:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:14,243 [237] [DEBUG] [app] Starting request: urn:request:6f485eff-9dff-4a3a-a450-64c1de51604e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:35:14,243 [236] [DEBUG] [app] Starting request: urn:request:e4525620-582a-4b44-aab0-2e8908c42137 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:35:14,243 [237] [DEBUG] [app] Ending request: urn:request:6f485eff-9dff-4a3a-a450-64c1de51604e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f485eff-9dff-4a3a-a450-64c1de51604e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:14,243 [236] [DEBUG] [app] Ending request: urn:request:e4525620-582a-4b44-aab0-2e8908c42137 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4525620-582a-4b44-aab0-2e8908c42137', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:35:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:35:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:35:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:35:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:14,255 [238] [DEBUG] [app] Ending request: urn:request:9dedb1b0-54aa-47b8-8032-f8c53032a6be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9dedb1b0-54aa-47b8-8032-f8c53032a6be', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:14,255 [235] [DEBUG] [app] Ending request: urn:request:b7da2bba-5415-4023-aa84-f34dae3b8c7c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b7da2bba-5415-4023-aa84-f34dae3b8c7c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:35:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:35:15,858 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:35:15,862 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:35:15,866 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:35:15,868 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:35:15,871 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:35:15,882 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:35:16,285 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:35:16,509 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:35:16,890 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:35:16,893 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:35:16,895 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:35:17,844 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:35:17,847 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:35:17,850 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:35:17,853 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:35:17,858 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:35:17,863 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:35:17,866 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:35:17,869 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:35:17,872 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:35:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:35:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:28.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 03:35:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:28 UTC)" (scheduled at 2025-02-07 03:35:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:35:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:35:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 18, 732851), True, datetime.datetime(2025, 2, 7, 3, 35, 18, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:35:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:35:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:35:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:35:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:35:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:03.419110+00:00 (in 39.997750 seconds)
repositorygcworker stdout | 2025-02-07 03:35:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:23 UTC)" (scheduled at 2025-02-07 03:35:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:35:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:35:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:35:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:35:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:40.424250+00:00 (in 16.000691 seconds)
gcworker stdout | 2025-02-07 03:35:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:54 UTC)" (scheduled at 2025-02-07 03:35:24.423055+00:00)
gcworker stdout | 2025-02-07 03:35:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:35:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:35:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:35:25,485 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:35:27,435 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:35:28,406 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:35:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:35:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:38.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 03:35:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:38 UTC)" (scheduled at 2025-02-07 03:35:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:35:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:35:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 28, 732872), True, datetime.datetime(2025, 2, 7, 3, 35, 28, 732872), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:35:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:35:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:35:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:35:29,236 [238] [DEBUG] [app] Starting request: urn:request:dafc2b97-e608-4e36-84d1-6c432de0f376 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:29,236 [235] [DEBUG] [app] Starting request: urn:request:f723dc82-e06c-451c-a2f2-ae7773c473aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:35:29,238 [246] [DEBUG] [app] Starting request: urn:request:1ae1f1a9-836e-4b20-9f8a-add797fe5218 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:29,238 [245] [DEBUG] [app] Starting request: urn:request:59616ed5-409b-4537-ae81-0b24e521461b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:29,239 [246] [DEBUG] [app] Ending request: urn:request:1ae1f1a9-836e-4b20-9f8a-add797fe5218 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1ae1f1a9-836e-4b20-9f8a-add797fe5218', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:35:29,239 [245] [DEBUG] [app] Ending request: urn:request:59616ed5-409b-4537-ae81-0b24e521461b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59616ed5-409b-4537-ae81-0b24e521461b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:35:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:35:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:35:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:29,241 [235] [DEBUG] [app] Starting request: urn:request:091c77e3-2b0c-4d05-b0b5-186fbaf9d226 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:35:29,242 [236] [DEBUG] [app] Starting request: urn:request:b6bc7255-0d30-4886-8fe3-a80271bb13fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:35:29,242 [235] [DEBUG] [app] Ending request: urn:request:091c77e3-2b0c-4d05-b0b5-186fbaf9d226 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:091c77e3-2b0c-4d05-b0b5-186fbaf9d226', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:29,242 [236] [DEBUG] [app] Ending request: urn:request:b6bc7255-0d30-4886-8fe3-a80271bb13fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6bc7255-0d30-4886-8fe3-a80271bb13fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:35:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:35:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:35:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:29,254 [235] [DEBUG] [app] Ending request: urn:request:f723dc82-e06c-451c-a2f2-ae7773c473aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f723dc82-e06c-451c-a2f2-ae7773c473aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:29,254 [238] [DEBUG] [app] Ending request: urn:request:dafc2b97-e608-4e36-84d1-6c432de0f376 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dafc2b97-e608-4e36-84d1-6c432de0f376', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:35:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:35:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:35:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:32.625945+00:00 (in 1.004332 seconds)
namespacegcworker stdout | 2025-02-07 03:35:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:31 UTC)" (scheduled at 2025-02-07 03:35:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:35:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:35:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 31, 621887), True, datetime.datetime(2025, 2, 7, 3, 35, 31, 621887), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:35:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:35:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:35:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:35:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:35:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:31.621131+00:00 (in 58.994688 seconds)
namespacegcworker stdout | 2025-02-07 03:35:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:32 UTC)" (scheduled at 2025-02-07 03:35:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:35:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:35:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:35:32,771 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:35:33,763 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:35:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:35:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:03.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 03:35:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:03 UTC)" (scheduled at 2025-02-07 03:35:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:35:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 35, 33, 919992), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:35:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:35:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:35:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:56.713547+00:00 (in 18.997209 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:37 UTC)" (scheduled at 2025-02-07 03:35:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:35:38,427 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:35:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:35:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:48.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 03:35:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:48 UTC)" (scheduled at 2025-02-07 03:35:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:35:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:35:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 38, 732871), True, datetime.datetime(2025, 2, 7, 3, 35, 38, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:35:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:35:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:35:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:35:38,957 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:35:40,129 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:35:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:35:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:54.423055+00:00 (in 13.998338 seconds)
gcworker stdout | 2025-02-07 03:35:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:10 UTC)" (scheduled at 2025-02-07 03:35:40.424250+00:00)
gcworker stdout | 2025-02-07 03:35:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:35:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899040430, None, 1, 0])
gcworker stdout | 2025-02-07 03:35:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:35:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:35:41,002 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:35:41,034 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:35:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:35:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:11.831521+00:00 (in 29.999530 seconds)
autopruneworker stdout | 2025-02-07 03:35:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:11 UTC)" (scheduled at 2025-02-07 03:35:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:35:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895741836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:35:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:35:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:35:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:35:42,121 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:42,469 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:35:43,193 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:35:43,247 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:35:43,310 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:35:43,674 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:35:44,163 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:35:44,211 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:35:44,236 [238] [DEBUG] [app] Starting request: urn:request:62bc6b66-52a5-4017-9ac2-3a6a2417c66b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:44,236 [235] [DEBUG] [app] Starting request: urn:request:15774583-8248-43bc-9d61-09873d1dfdff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:35:44,239 [245] [DEBUG] [app] Starting request: urn:request:094b6765-8079-49aa-819c-0bd03db7c47d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:44,239 [239] [DEBUG] [app] Starting request: urn:request:1461b85c-264b-447d-89f0-a7c77e3c86cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:44,239 [245] [DEBUG] [app] Ending request: urn:request:094b6765-8079-49aa-819c-0bd03db7c47d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:094b6765-8079-49aa-819c-0bd03db7c47d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:35:44,239 [239] [DEBUG] [app] Ending request: urn:request:1461b85c-264b-447d-89f0-a7c77e3c86cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1461b85c-264b-447d-89f0-a7c77e3c86cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:35:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:35:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:35:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:44,242 [235] [DEBUG] [app] Starting request: urn:request:ad478cc3-8ee7-4060-9dbe-1c9cbbd1f515 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:35:44,242 [235] [DEBUG] [app] Ending request: urn:request:ad478cc3-8ee7-4060-9dbe-1c9cbbd1f515 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad478cc3-8ee7-4060-9dbe-1c9cbbd1f515', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:44,242 [236] [DEBUG] [app] Starting request: urn:request:fc729772-f49f-4d89-b7da-73b2a55f8b91 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:35:44,242 [236] [DEBUG] [app] Ending request: urn:request:fc729772-f49f-4d89-b7da-73b2a55f8b91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc729772-f49f-4d89-b7da-73b2a55f8b91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:35:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:35:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
securityscanningnotificationworker stdout | 2025-02-07 03:35:44,251 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:35:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:44,254 [238] [DEBUG] [app] Ending request: urn:request:62bc6b66-52a5-4017-9ac2-3a6a2417c66b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62bc6b66-52a5-4017-9ac2-3a6a2417c66b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:44,254 [235] [DEBUG] [app] Ending request: urn:request:15774583-8248-43bc-9d61-09873d1dfdff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15774583-8248-43bc-9d61-09873d1dfdff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:35:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:35:45,865 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:35:45,868 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:35:45,872 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:35:45,876 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:35:45,878 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:35:45,915 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:35:46,322 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:35:46,518 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:35:46,899 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:35:46,903 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:35:46,906 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:35:47,854 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:35:47,857 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:35:47,861 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:35:47,864 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:35:47,867 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:35:47,872 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:35:47,877 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:35:47,880 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:35:47,882 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:35:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:35:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:50.814677+00:00 (in 2.082092 seconds)
notificationworker stdout | 2025-02-07 03:35:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:58 UTC)" (scheduled at 2025-02-07 03:35:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:35:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:35:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 48, 732870), True, datetime.datetime(2025, 2, 7, 3, 35, 48, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:35:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:35:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:35:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:35:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:35:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:35:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:35:58.732147+00:00 (in 7.917017 seconds)
notificationworker stdout | 2025-02-07 03:35:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:50 UTC)" (scheduled at 2025-02-07 03:35:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:35:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:35:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:35:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:35:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:10.424250+00:00 (in 16.000674 seconds)
gcworker stdout | 2025-02-07 03:35:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:24 UTC)" (scheduled at 2025-02-07 03:35:54.423055+00:00)
gcworker stdout | 2025-02-07 03:35:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:35:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:35:55,493 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:35:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:35:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:56.118855+00:00 (in 59.999505 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:35:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:56 UTC)" (scheduled at 2025-02-07 03:35:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:35:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:35:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:35:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:37.715878+00:00 (in 41.001864 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:56 UTC)" (scheduled at 2025-02-07 03:35:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:35:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:35:57,445 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:35:58,441 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:35:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:35:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:08.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 03:35:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:08 UTC)" (scheduled at 2025-02-07 03:35:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:35:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:35:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 35, 58, 732873), True, datetime.datetime(2025, 2, 7, 3, 35, 58, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:35:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:35:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:35:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:35:59,235 [237] [DEBUG] [app] Starting request: urn:request:ba72f71d-d9c1-40e6-be07-9c196334ab13 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:59,235 [238] [DEBUG] [app] Starting request: urn:request:230a2c2f-e95b-4f08-b883-22c7977957b3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:35:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:35:59,238 [245] [DEBUG] [app] Starting request: urn:request:7899ac3a-0ea5-4242-b128-e188f4ef91e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:59,238 [246] [DEBUG] [app] Starting request: urn:request:041d4ccb-c9b7-46c2-8097-80e7252808ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:35:59,238 [245] [DEBUG] [app] Ending request: urn:request:7899ac3a-0ea5-4242-b128-e188f4ef91e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7899ac3a-0ea5-4242-b128-e188f4ef91e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:35:59,238 [246] [DEBUG] [app] Ending request: urn:request:041d4ccb-c9b7-46c2-8097-80e7252808ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:041d4ccb-c9b7-46c2-8097-80e7252808ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:35:59,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:35:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:35:59,241 [238] [DEBUG] [app] Starting request: urn:request:2ec2158f-ce4b-409a-9d9b-41525e8f527f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:35:59,241 [238] [DEBUG] [app] Ending request: urn:request:2ec2158f-ce4b-409a-9d9b-41525e8f527f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2ec2158f-ce4b-409a-9d9b-41525e8f527f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:59,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:59,241 [236] [DEBUG] [app] Starting request: urn:request:d91a503e-dd27-4ecf-9aeb-0ba2ba06b906 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:35:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:35:59,242 [236] [DEBUG] [app] Ending request: urn:request:d91a503e-dd27-4ecf-9aeb-0ba2ba06b906 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d91a503e-dd27-4ecf-9aeb-0ba2ba06b906', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:35:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:35:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:35:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:35:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:35:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:35:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:35:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:35:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:35:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:35:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:35:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:35:59,253 [238] [DEBUG] [app] Ending request: urn:request:230a2c2f-e95b-4f08-b883-22c7977957b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:230a2c2f-e95b-4f08-b883-22c7977957b3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:59,253 [237] [DEBUG] [app] Ending request: urn:request:ba72f71d-d9c1-40e6-be07-9c196334ab13 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba72f71d-d9c1-40e6-be07-9c196334ab13', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:35:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:35:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:35:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:35:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:35:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:35:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:35:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:04.936688+00:00 (in 4.998314 seconds)
exportactionlogsworker stdout | 2025-02-07 03:35:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:59 UTC)" (scheduled at 2025-02-07 03:35:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:35:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:35:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:36:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:36:02,808 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:36:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:36:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:23.420886+00:00 (in 20.001316 seconds)
repositorygcworker stdout | 2025-02-07 03:36:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:03 UTC)" (scheduled at 2025-02-07 03:36:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:36:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:36:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 3, 419844), True, datetime.datetime(2025, 2, 7, 3, 36, 3, 419844), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:36:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:36:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:36:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:36:03,799 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:36:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:36:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:33.919286+00:00 (in 29.999543 seconds)
buildlogsarchiver stdout | 2025-02-07 03:36:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:33 UTC)" (scheduled at 2025-02-07 03:36:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:36:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 36, 3, 920001), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:36:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:36:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:36:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:36:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:36:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:59.937899+00:00 (in 55.000759 seconds)
exportactionlogsworker stdout | 2025-02-07 03:36:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:04 UTC)" (scheduled at 2025-02-07 03:36:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:36:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:36:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 4, 937394), True, datetime.datetime(2025, 2, 7, 3, 36, 4, 937394), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:36:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:36:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:36:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:36:08,463 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:36:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:36:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:18.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 03:36:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:18 UTC)" (scheduled at 2025-02-07 03:36:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:36:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:36:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 8, 732875), True, datetime.datetime(2025, 2, 7, 3, 36, 8, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:36:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:36:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:36:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:36:08,967 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:36:10,140 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:36:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:36:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:24.423055+00:00 (in 13.998362 seconds)
gcworker stdout | 2025-02-07 03:36:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:40 UTC)" (scheduled at 2025-02-07 03:36:10.424250+00:00)
gcworker stdout | 2025-02-07 03:36:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:36:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899070430, None, 1, 0])
gcworker stdout | 2025-02-07 03:36:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:36:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:36:11,018 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:36:11,070 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:36:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:36:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:41.831521+00:00 (in 29.999520 seconds)
autopruneworker stdout | 2025-02-07 03:36:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:41 UTC)" (scheduled at 2025-02-07 03:36:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:36:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895771836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:36:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:36:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:36:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:36:12,154 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:12,505 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:36:13,211 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:36:13,283 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:36:13,343 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:36:13,689 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:36:14,193 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:36:14,230 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:36:14,236 [238] [DEBUG] [app] Starting request: urn:request:936dcfb8-4d3c-4b6e-8201-1afcd0af320c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:14,236 [236] [DEBUG] [app] Starting request: urn:request:01fd8226-5d44-4a8b-b0e2-92de0985e62d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:36:14,238 [246] [DEBUG] [app] Starting request: urn:request:23a310a7-ab6a-41cb-90a0-40f5ab40eb98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:14,239 [239] [DEBUG] [app] Starting request: urn:request:899c9bb0-03fa-4aae-8c65-2dae4d64ea58 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:14,239 [246] [DEBUG] [app] Ending request: urn:request:23a310a7-ab6a-41cb-90a0-40f5ab40eb98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23a310a7-ab6a-41cb-90a0-40f5ab40eb98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:14,239 [239] [DEBUG] [app] Ending request: urn:request:899c9bb0-03fa-4aae-8c65-2dae4d64ea58 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:899c9bb0-03fa-4aae-8c65-2dae4d64ea58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:36:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:36:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:14,242 [235] [DEBUG] [app] Starting request: urn:request:8c2320de-d127-45e6-95af-83bbfa1fa7d5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:36:14,242 [235] [DEBUG] [app] Ending request: urn:request:8c2320de-d127-45e6-95af-83bbfa1fa7d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c2320de-d127-45e6-95af-83bbfa1fa7d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:36:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:36:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:14,243 [237] [DEBUG] [app] Starting request: urn:request:e60617f1-f17d-4719-be26-40e5decac51d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:36:14,243 [237] [DEBUG] [app] Ending request: urn:request:e60617f1-f17d-4719-be26-40e5decac51d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e60617f1-f17d-4719-be26-40e5decac51d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:36:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:36:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:14,254 [236] [DEBUG] [app] Ending request: urn:request:01fd8226-5d44-4a8b-b0e2-92de0985e62d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01fd8226-5d44-4a8b-b0e2-92de0985e62d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:36:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:36:14,254 [238] [DEBUG] [app] Ending request: urn:request:936dcfb8-4d3c-4b6e-8201-1afcd0af320c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:936dcfb8-4d3c-4b6e-8201-1afcd0af320c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:36:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
securityscanningnotificationworker stdout | 2025-02-07 03:36:14,287 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:36:15,876 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:36:15,879 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:36:15,884 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:36:15,887 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:36:15,890 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:36:15,937 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:36:16,336 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:36:16,551 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:36:16,908 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:36:16,910 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:36:16,913 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:36:17,863 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:36:17,866 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:36:17,870 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:36:17,872 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:36:17,876 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:36:17,879 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:36:17,883 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:36:17,887 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:36:17,893 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:36:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:36:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:28.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 03:36:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:28 UTC)" (scheduled at 2025-02-07 03:36:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:36:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:36:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 18, 732918), True, datetime.datetime(2025, 2, 7, 3, 36, 18, 732918), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:36:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:36:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:36:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:36:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:36:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:03.419110+00:00 (in 39.997742 seconds)
repositorygcworker stdout | 2025-02-07 03:36:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:23 UTC)" (scheduled at 2025-02-07 03:36:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:36:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:36:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:36:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:36:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:40.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 03:36:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:54 UTC)" (scheduled at 2025-02-07 03:36:24.423055+00:00)
gcworker stdout | 2025-02-07 03:36:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:36:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:36:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:36:25,529 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:36:27,468 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:36:28,475 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:36:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:36:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:38.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 03:36:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:38 UTC)" (scheduled at 2025-02-07 03:36:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:36:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:36:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 28, 732891), True, datetime.datetime(2025, 2, 7, 3, 36, 28, 732891), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:36:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:36:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:36:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:36:29,235 [235] [DEBUG] [app] Starting request: urn:request:c1008ec3-7e8a-4475-8c83-be455d9970cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:29,235 [238] [DEBUG] [app] Starting request: urn:request:ff66aafc-cd2d-4a4f-88e4-f0732cc03661 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:36:29,239 [239] [DEBUG] [app] Starting request: urn:request:0eddb392-0f61-4809-9fa5-4080081a6d42 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:29,239 [246] [DEBUG] [app] Starting request: urn:request:50c1d674-7731-476c-97ab-53a3de15db33 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:29,239 [239] [DEBUG] [app] Ending request: urn:request:0eddb392-0f61-4809-9fa5-4080081a6d42 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0eddb392-0f61-4809-9fa5-4080081a6d42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:29,239 [246] [DEBUG] [app] Ending request: urn:request:50c1d674-7731-476c-97ab-53a3de15db33 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50c1d674-7731-476c-97ab-53a3de15db33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:36:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:36:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:29,242 [235] [DEBUG] [app] Starting request: urn:request:f37b44c6-5116-40a7-b543-c01d67b221a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:36:29,242 [235] [DEBUG] [app] Ending request: urn:request:f37b44c6-5116-40a7-b543-c01d67b221a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f37b44c6-5116-40a7-b543-c01d67b221a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:36:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:36:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:29,243 [237] [DEBUG] [app] Starting request: urn:request:33790a7c-112f-4186-8e15-f4f97b366de6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:36:29,243 [237] [DEBUG] [app] Ending request: urn:request:33790a7c-112f-4186-8e15-f4f97b366de6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:33790a7c-112f-4186-8e15-f4f97b366de6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:36:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:36:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:29,255 [235] [DEBUG] [app] Ending request: urn:request:c1008ec3-7e8a-4475-8c83-be455d9970cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c1008ec3-7e8a-4475-8c83-be455d9970cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:36:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 03:36:29,256 [238] [DEBUG] [app] Ending request: urn:request:ff66aafc-cd2d-4a4f-88e4-f0732cc03661 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ff66aafc-cd2d-4a4f-88e4-f0732cc03661', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:36:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:36:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:36:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:32.625945+00:00 (in 1.004367 seconds)
namespacegcworker stdout | 2025-02-07 03:36:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:31 UTC)" (scheduled at 2025-02-07 03:36:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:36:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:36:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 31, 621852), True, datetime.datetime(2025, 2, 7, 3, 36, 31, 621852), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:36:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:36:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:36:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:36:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:36:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:13.624301+00:00 (in 40.997923 seconds)
namespacegcworker stdout | 2025-02-07 03:36:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:32 UTC)" (scheduled at 2025-02-07 03:36:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:36:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:36:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:36:32,818 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:36:33,831 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:36:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:36:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:03.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 03:36:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:03 UTC)" (scheduled at 2025-02-07 03:36:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:36:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 36, 33, 919981), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:36:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:36:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:36:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:56.713547+00:00 (in 18.997198 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:37 UTC)" (scheduled at 2025-02-07 03:36:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:36:38,501 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:36:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:36:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:48.732147+00:00 (in 9.999452 seconds)
notificationworker stdout | 2025-02-07 03:36:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:48 UTC)" (scheduled at 2025-02-07 03:36:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:36:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:36:38,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 38, 733120), True, datetime.datetime(2025, 2, 7, 3, 36, 38, 733120), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:36:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:36:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:36:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:36:39,003 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:36:40,170 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:36:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:36:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:54.423055+00:00 (in 13.998357 seconds)
gcworker stdout | 2025-02-07 03:36:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:10 UTC)" (scheduled at 2025-02-07 03:36:40.424250+00:00)
gcworker stdout | 2025-02-07 03:36:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:36:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899100430, None, 1, 0])
gcworker stdout | 2025-02-07 03:36:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:36:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:36:41,055 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:36:41,106 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:36:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:36:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:11.831521+00:00 (in 29.999474 seconds)
autopruneworker stdout | 2025-02-07 03:36:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:11 UTC)" (scheduled at 2025-02-07 03:36:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:36:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895801836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:36:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:36:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:36:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:36:42,190 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:42,542 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:36:43,218 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:36:43,320 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:36:43,353 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:36:43,726 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:36:44,220 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-web stdout | 2025-02-07 03:36:44,236 [237] [DEBUG] [app] Starting request: urn:request:2ec6db93-c211-4a21-ba0f-5d0838d08ef8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:44,236 [238] [DEBUG] [app] Starting request: urn:request:795eaead-58c9-48ce-a1e4-24adb484af3c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:44,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:44,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:36:44,240 [246] [DEBUG] [app] Starting request: urn:request:1b7d1bc5-befc-45c1-96d0-8a441a977101 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:44,240 [239] [DEBUG] [app] Starting request: urn:request:618c54c1-3c12-40ef-8623-294e8146ea8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:44,240 [246] [DEBUG] [app] Ending request: urn:request:1b7d1bc5-befc-45c1-96d0-8a441a977101 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b7d1bc5-befc-45c1-96d0-8a441a977101', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:44,241 [239] [DEBUG] [app] Ending request: urn:request:618c54c1-3c12-40ef-8623-294e8146ea8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:618c54c1-3c12-40ef-8623-294e8146ea8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:44,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:36:44,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:36:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:44,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:44,243 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:44,244 [238] [DEBUG] [app] Starting request: urn:request:0dd51dde-d103-45d2-a252-d6b8fe63693c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:36:44,244 [236] [DEBUG] [app] Starting request: urn:request:e5a6deb5-ca08-4203-8e22-bf665cec8789 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:36:44,245 [238] [DEBUG] [app] Ending request: urn:request:0dd51dde-d103-45d2-a252-d6b8fe63693c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0dd51dde-d103-45d2-a252-d6b8fe63693c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:36:44,245 [236] [DEBUG] [app] Ending request: urn:request:e5a6deb5-ca08-4203-8e22-bf665cec8789 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5a6deb5-ca08-4203-8e22-bf665cec8789', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:36:44,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:36:44,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:36:44,245 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:44,246 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:44,246 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:44,246 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:44,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:44,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:44,250 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:44,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:44,250 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:44,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:44,255 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:44,255 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:44,256 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:44,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:44,257 [237] [DEBUG] [app] Ending request: urn:request:2ec6db93-c211-4a21-ba0f-5d0838d08ef8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2ec6db93-c211-4a21-ba0f-5d0838d08ef8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:44,258 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:36:44,258 [238] [DEBUG] [app] Ending request: urn:request:795eaead-58c9-48ce-a1e4-24adb484af3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:795eaead-58c9-48ce-a1e4-24adb484af3c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:44,258 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:36:44,258 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 03:36:44,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
manifestbackfillworker stdout | 2025-02-07 03:36:44,267 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:36:44,325 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:36:45,884 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:36:45,887 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:36:45,891 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:36:45,894 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:36:45,896 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:36:45,947 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:36:46,344 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:36:46,587 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:36:46,915 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:36:46,919 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:36:46,922 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:36:47,871 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:36:47,873 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:36:47,877 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:36:47,883 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:36:47,886 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:36:47,888 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:36:47,893 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:36:47,896 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:36:47,902 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:36:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:36:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:50.814677+00:00 (in 2.082072 seconds)
notificationworker stdout | 2025-02-07 03:36:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:58 UTC)" (scheduled at 2025-02-07 03:36:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:36:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:36:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 48, 732887), True, datetime.datetime(2025, 2, 7, 3, 36, 48, 732887), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:36:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:36:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:36:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:36:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:36:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:36:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:36:58.732147+00:00 (in 7.917031 seconds)
notificationworker stdout | 2025-02-07 03:36:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:50 UTC)" (scheduled at 2025-02-07 03:36:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:36:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:36:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:36:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:36:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 03:36:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:24 UTC)" (scheduled at 2025-02-07 03:36:54.423055+00:00)
gcworker stdout | 2025-02-07 03:36:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:36:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:36:55,555 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:36:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:36:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:56.118855+00:00 (in 59.999501 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:36:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:56 UTC)" (scheduled at 2025-02-07 03:36:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:36:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:36:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:36:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:37.715878+00:00 (in 41.001880 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:56 UTC)" (scheduled at 2025-02-07 03:36:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:36:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:36:57,504 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:36:58,511 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:36:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:36:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:08.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 03:36:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:08 UTC)" (scheduled at 2025-02-07 03:36:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:36:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:36:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 36, 58, 732909), True, datetime.datetime(2025, 2, 7, 3, 36, 58, 732909), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:36:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:36:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:36:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:36:59,235 [238] [DEBUG] [app] Starting request: urn:request:a80fc58c-70e9-4064-92a3-c163bec788a9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:59,235 [236] [DEBUG] [app] Starting request: urn:request:7fde6dfe-e962-461c-82ab-6361098b137e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:36:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:36:59,238 [239] [DEBUG] [app] Starting request: urn:request:25c5ce95-95cb-4aec-94a1-15610e62b10f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:59,239 [246] [DEBUG] [app] Starting request: urn:request:4abba7e8-37ac-4c85-91dc-98e7642936ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:36:59,239 [239] [DEBUG] [app] Ending request: urn:request:25c5ce95-95cb-4aec-94a1-15610e62b10f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:25c5ce95-95cb-4aec-94a1-15610e62b10f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:59,239 [246] [DEBUG] [app] Ending request: urn:request:4abba7e8-37ac-4c85-91dc-98e7642936ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4abba7e8-37ac-4c85-91dc-98e7642936ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:36:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:36:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:36:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:36:59,242 [238] [DEBUG] [app] Starting request: urn:request:faca6a15-370d-4adc-84aa-d13204df2c70 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:36:59,242 [238] [DEBUG] [app] Ending request: urn:request:faca6a15-370d-4adc-84aa-d13204df2c70 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:faca6a15-370d-4adc-84aa-d13204df2c70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:36:59,242 [237] [DEBUG] [app] Starting request: urn:request:61af5fd3-3f0d-41dd-ab35-122299699536 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:36:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:36:59,243 [237] [DEBUG] [app] Ending request: urn:request:61af5fd3-3f0d-41dd-ab35-122299699536 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61af5fd3-3f0d-41dd-ab35-122299699536', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:36:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:36:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:36:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:36:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:36:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:36:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:36:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:36:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:36:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:36:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:36:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:36:59,255 [236] [DEBUG] [app] Ending request: urn:request:7fde6dfe-e962-461c-82ab-6361098b137e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7fde6dfe-e962-461c-82ab-6361098b137e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:59,255 [238] [DEBUG] [app] Ending request: urn:request:a80fc58c-70e9-4064-92a3-c163bec788a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a80fc58c-70e9-4064-92a3-c163bec788a9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:36:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:36:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:36:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:36:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:36:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:36:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 03:36:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:36:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:04.936688+00:00 (in 4.998289 seconds)
exportactionlogsworker stdout | 2025-02-07 03:36:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:59 UTC)" (scheduled at 2025-02-07 03:36:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:36:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:36:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:37:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:37:02,853 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:37:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:37:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:23.420886+00:00 (in 20.001325 seconds)
repositorygcworker stdout | 2025-02-07 03:37:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:03 UTC)" (scheduled at 2025-02-07 03:37:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:37:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:37:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 3, 419838), True, datetime.datetime(2025, 2, 7, 3, 37, 3, 419838), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:37:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:37:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:37:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:37:03,867 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:37:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:37:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:33.919286+00:00 (in 29.999527 seconds)
buildlogsarchiver stdout | 2025-02-07 03:37:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:33 UTC)" (scheduled at 2025-02-07 03:37:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:37:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 37, 3, 920040), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:37:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:37:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:37:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:37:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:37:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:59.937899+00:00 (in 55.000748 seconds)
exportactionlogsworker stdout | 2025-02-07 03:37:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:04 UTC)" (scheduled at 2025-02-07 03:37:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:37:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:37:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 4, 937503), True, datetime.datetime(2025, 2, 7, 3, 37, 4, 937503), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:37:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:37:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:37:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:37:08,511 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:37:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:18.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 03:37:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:18 UTC)" (scheduled at 2025-02-07 03:37:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:37:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:37:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 8, 732880), True, datetime.datetime(2025, 2, 7, 3, 37, 8, 732880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:37:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:37:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:37:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:37:09,040 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:37:10,207 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:37:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:37:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:24.423055+00:00 (in 13.998180 seconds)
gcworker stdout | 2025-02-07 03:37:10,425 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:40 UTC)" (scheduled at 2025-02-07 03:37:10.424250+00:00)
gcworker stdout | 2025-02-07 03:37:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:37:10,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899130431, None, 1, 0])
gcworker stdout | 2025-02-07 03:37:10,434 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:37:10,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:37:11,091 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:37:11,143 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:37:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:37:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:41.831521+00:00 (in 29.999513 seconds)
autopruneworker stdout | 2025-02-07 03:37:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:41 UTC)" (scheduled at 2025-02-07 03:37:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:37:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895831836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:37:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:37:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:37:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:37:12,226 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:12,566 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:37:13,227 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:37:13,334 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:37:13,366 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
namespacegcworker stdout | 2025-02-07 03:37:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:37:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:31.621131+00:00 (in 17.996427 seconds)
namespacegcworker stdout | 2025-02-07 03:37:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:42:13 UTC)" (scheduled at 2025-02-07 03:37:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:37:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 37, 13, 624979), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:37:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 37, 13, 624979), True, datetime.datetime(2025, 2, 7, 3, 37, 13, 624979), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:37:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 37, 13, 624979), True, datetime.datetime(2025, 2, 7, 3, 37, 13, 624979), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 37, 13, 624979), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:37:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:37:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:42:13 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:37:13,747 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:37:14,236 [237] [DEBUG] [app] Starting request: urn:request:b9aa8f82-9785-46db-9a6e-d7798b6194cf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:14,236 [236] [DEBUG] [app] Starting request: urn:request:5baacdc6-731b-4a28-b9db-a46d29eb7209 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:37:14,239 [239] [DEBUG] [app] Starting request: urn:request:6d01eb0d-46b2-4628-822b-b8dc7d2c199a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:14,239 [244] [DEBUG] [app] Starting request: urn:request:9ab150a2-1498-4abd-afcf-a7a1e1e4de5a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:14,239 [244] [DEBUG] [app] Ending request: urn:request:9ab150a2-1498-4abd-afcf-a7a1e1e4de5a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9ab150a2-1498-4abd-afcf-a7a1e1e4de5a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:37:14,239 [239] [DEBUG] [app] Ending request: urn:request:6d01eb0d-46b2-4628-822b-b8dc7d2c199a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d01eb0d-46b2-4628-822b-b8dc7d2c199a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:37:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:37:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:37:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:14,242 [235] [DEBUG] [app] Starting request: urn:request:22bf8bb7-29de-4989-92fa-d8b77328708f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:14,242 [237] [DEBUG] [app] Starting request: urn:request:d70122f6-d9f0-4800-8739-be1dcb01cfad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:14,242 [235] [DEBUG] [app] Ending request: urn:request:22bf8bb7-29de-4989-92fa-d8b77328708f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:22bf8bb7-29de-4989-92fa-d8b77328708f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:37:14,242 [237] [DEBUG] [app] Ending request: urn:request:d70122f6-d9f0-4800-8739-be1dcb01cfad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d70122f6-d9f0-4800-8739-be1dcb01cfad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:37:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:37:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:37:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:14,254 [237] [DEBUG] [app] Ending request: urn:request:b9aa8f82-9785-46db-9a6e-d7798b6194cf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b9aa8f82-9785-46db-9a6e-d7798b6194cf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:37:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:37:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:37:14,255 [236] [DEBUG] [app] Ending request: urn:request:5baacdc6-731b-4a28-b9db-a46d29eb7209 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5baacdc6-731b-4a28-b9db-a46d29eb7209', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:37:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:37:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gcworker stdout | 2025-02-07 03:37:14,257 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:37:14,303 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:37:14,361 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:37:15,892 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:37:15,895 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:37:15,898 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:37:15,900 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:37:15,903 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:37:15,984 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:37:16,376 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:37:16,597 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:37:16,923 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:37:16,927 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:37:16,929 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:37:17,878 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:37:17,885 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:37:17,888 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:37:17,894 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:37:17,897 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:37:17,900 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:37:17,904 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:37:17,906 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:37:17,911 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:37:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:28.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 03:37:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:28 UTC)" (scheduled at 2025-02-07 03:37:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:37:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:37:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 18, 732903), True, datetime.datetime(2025, 2, 7, 3, 37, 18, 732903), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:37:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:37:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:37:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:37:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:37:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:03.419110+00:00 (in 39.997700 seconds)
repositorygcworker stdout | 2025-02-07 03:37:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:23 UTC)" (scheduled at 2025-02-07 03:37:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:37:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:37:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:37:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:37:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:40.424250+00:00 (in 16.000696 seconds)
gcworker stdout | 2025-02-07 03:37:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:54 UTC)" (scheduled at 2025-02-07 03:37:24.423055+00:00)
gcworker stdout | 2025-02-07 03:37:24,423 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 03:37:24,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:37:24,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:37:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:37:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:37:25,589 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:37:27,540 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:37:28,547 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:37:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:33.814166+00:00 (in 5.081586 seconds)
notificationworker stdout | 2025-02-07 03:37:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:38 UTC)" (scheduled at 2025-02-07 03:37:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:37:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:37:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 28, 732848), True, datetime.datetime(2025, 2, 7, 3, 37, 28, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:37:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:37:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:37:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:37:29,236 [235] [DEBUG] [app] Starting request: urn:request:c6c29466-3ec9-4088-a78a-793cd5e42c46 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:29,236 [238] [DEBUG] [app] Starting request: urn:request:e6f50b14-1130-4980-a6dd-65f0959702b1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:37:29,238 [239] [DEBUG] [app] Starting request: urn:request:1788a9ca-aa79-447f-a470-b4e405e2b3ed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:29,239 [239] [DEBUG] [app] Ending request: urn:request:1788a9ca-aa79-447f-a470-b4e405e2b3ed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1788a9ca-aa79-447f-a470-b4e405e2b3ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:37:29,239 [246] [DEBUG] [app] Starting request: urn:request:59d9af95-df43-424c-a37e-fc25c06dd5c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:37:29,239 [246] [DEBUG] [app] Ending request: urn:request:59d9af95-df43-424c-a37e-fc25c06dd5c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59d9af95-df43-424c-a37e-fc25c06dd5c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:37:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:37:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:37:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:29,242 [236] [DEBUG] [app] Starting request: urn:request:20c7258b-1ed3-4e58-8d47-8cc14093b04d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:29,242 [236] [DEBUG] [app] Ending request: urn:request:20c7258b-1ed3-4e58-8d47-8cc14093b04d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:20c7258b-1ed3-4e58-8d47-8cc14093b04d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:37:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:37:29,243 [238] [DEBUG] [app] Starting request: urn:request:860e2507-c0ff-4721-8698-07be139d1424 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:29,243 [238] [DEBUG] [app] Ending request: urn:request:860e2507-c0ff-4721-8698-07be139d1424 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:860e2507-c0ff-4721-8698-07be139d1424', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:37:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:37:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:29,255 [235] [DEBUG] [app] Ending request: urn:request:c6c29466-3ec9-4088-a78a-793cd5e42c46 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c6c29466-3ec9-4088-a78a-793cd5e42c46', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:37:29,255 [238] [DEBUG] [app] Ending request: urn:request:e6f50b14-1130-4980-a6dd-65f0959702b1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6f50b14-1130-4980-a6dd-65f0959702b1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:37:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:37:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:37:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:37:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:37:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:37:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:32.625945+00:00 (in 1.004363 seconds)
namespacegcworker stdout | 2025-02-07 03:37:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:31 UTC)" (scheduled at 2025-02-07 03:37:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:37:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:37:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 31, 621857), True, datetime.datetime(2025, 2, 7, 3, 37, 31, 621857), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:37:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:37:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:37:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:37:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:37:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:31.621131+00:00 (in 58.994702 seconds)
namespacegcworker stdout | 2025-02-07 03:37:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:32 UTC)" (scheduled at 2025-02-07 03:37:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:37:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:37:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:37:32,873 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 03:37:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:38.732147+00:00 (in 4.917523 seconds)
notificationworker stdout | 2025-02-07 03:37:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:42:33 UTC)" (scheduled at 2025-02-07 03:37:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:37:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 37, 33, 814894), 'notification/%'])
notificationworker stdout | 2025-02-07 03:37:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 37, 33, 814894), True, datetime.datetime(2025, 2, 7, 3, 37, 33, 814894), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:37:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 37, 33, 814894), True, datetime.datetime(2025, 2, 7, 3, 37, 33, 814894), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 37, 33, 814894), 'notification/%'])
notificationworker stdout | 2025-02-07 03:37:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:37:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:42:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:37:33,887 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:37:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:37:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:03.919286+00:00 (in 29.999533 seconds)
buildlogsarchiver stdout | 2025-02-07 03:37:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:03 UTC)" (scheduled at 2025-02-07 03:37:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:37:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 37, 33, 920023), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:37:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:37:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:37:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:56.713547+00:00 (in 18.997207 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:37 UTC)" (scheduled at 2025-02-07 03:37:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:37:38,532 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:37:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:48.732147+00:00 (in 9.999530 seconds)
notificationworker stdout | 2025-02-07 03:37:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:48 UTC)" (scheduled at 2025-02-07 03:37:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:37:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:37:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 38, 732919), True, datetime.datetime(2025, 2, 7, 3, 37, 38, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:37:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:37:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:37:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:37:39,077 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:37:40,244 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:37:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:37:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:54.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 03:37:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:10 UTC)" (scheduled at 2025-02-07 03:37:40.424250+00:00)
gcworker stdout | 2025-02-07 03:37:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:37:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899160430, None, 1, 0])
gcworker stdout | 2025-02-07 03:37:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:37:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:37:41,128 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:37:41,179 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:37:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:37:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:11.831521+00:00 (in 29.999558 seconds)
autopruneworker stdout | 2025-02-07 03:37:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:11 UTC)" (scheduled at 2025-02-07 03:37:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:37:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895861836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:37:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:37:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:37:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:37:42,262 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:42,594 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:37:43,248 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:37:43,350 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:37:43,379 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:37:43,783 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:37:44,235 [238] [DEBUG] [app] Starting request: urn:request:aded810f-b85b-46ba-b830-e633965c3fc3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:44,235 [237] [DEBUG] [app] Starting request: urn:request:cd1dff90-1a37-49d2-9c44-6445724d58e8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:37:44,238 [246] [DEBUG] [app] Starting request: urn:request:acb1adfc-ea41-4623-81b5-371dadde1aec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:44,238 [239] [DEBUG] [app] Starting request: urn:request:0b8edee1-01ec-41b9-ab86-d2e26b2db21a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:44,239 [239] [DEBUG] [app] Ending request: urn:request:0b8edee1-01ec-41b9-ab86-d2e26b2db21a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b8edee1-01ec-41b9-ab86-d2e26b2db21a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:37:44,239 [246] [DEBUG] [app] Ending request: urn:request:acb1adfc-ea41-4623-81b5-371dadde1aec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:acb1adfc-ea41-4623-81b5-371dadde1aec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:37:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:37:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:37:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:44,243 [235] [DEBUG] [app] Starting request: urn:request:e28237a1-6502-474e-8ca7-8a91fad3de60 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:44,243 [236] [DEBUG] [app] Starting request: urn:request:57c31ee9-04dd-4ae3-8410-85662c8784fb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:44,244 [236] [DEBUG] [app] Ending request: urn:request:57c31ee9-04dd-4ae3-8410-85662c8784fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57c31ee9-04dd-4ae3-8410-85662c8784fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:37:44,244 [235] [DEBUG] [app] Ending request: urn:request:e28237a1-6502-474e-8ca7-8a91fad3de60 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e28237a1-6502-474e-8ca7-8a91fad3de60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:37:44,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:37:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:37:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:44,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:44,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:44,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:44,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:44,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:44,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:44,256 [237] [DEBUG] [app] Ending request: urn:request:cd1dff90-1a37-49d2-9c44-6445724d58e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd1dff90-1a37-49d2-9c44-6445724d58e8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:37:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:37:44,256 [238] [DEBUG] [app] Ending request: urn:request:aded810f-b85b-46ba-b830-e633965c3fc3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aded810f-b85b-46ba-b830-e633965c3fc3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:37:44,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:37:44,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 03:37:44,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gcworker stdout | 2025-02-07 03:37:44,282 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:37:44,318 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:37:44,388 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:37:45,901 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:37:45,905 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:37:45,908 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:37:45,911 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:37:45,913 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:37:46,019 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:37:46,388 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:37:46,624 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:37:46,931 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:37:46,935 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:37:46,937 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:37:47,890 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:37:47,895 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:37:47,897 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:37:47,904 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:37:47,909 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:37:47,912 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:37:47,915 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:37:47,918 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:37:47,920 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:37:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:50.814677+00:00 (in 2.082031 seconds)
notificationworker stdout | 2025-02-07 03:37:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:58 UTC)" (scheduled at 2025-02-07 03:37:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:37:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:37:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 48, 732970), True, datetime.datetime(2025, 2, 7, 3, 37, 48, 732970), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:37:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:37:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:37:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:37:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:37:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:37:58.732147+00:00 (in 7.916906 seconds)
notificationworker stdout | 2025-02-07 03:37:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:50 UTC)" (scheduled at 2025-02-07 03:37:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:37:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:37:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:37:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:37:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:10.424250+00:00 (in 16.000701 seconds)
gcworker stdout | 2025-02-07 03:37:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:24 UTC)" (scheduled at 2025-02-07 03:37:54.423055+00:00)
gcworker stdout | 2025-02-07 03:37:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:37:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:37:55,620 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:37:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:37:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:56.118855+00:00 (in 59.999466 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:37:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:56 UTC)" (scheduled at 2025-02-07 03:37:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:37:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:37:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:37:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:37.715878+00:00 (in 41.001842 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:56 UTC)" (scheduled at 2025-02-07 03:37:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:37:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:37:57,576 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:37:58,561 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:37:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:37:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:08.732147+00:00 (in 9.999478 seconds)
notificationworker stdout | 2025-02-07 03:37:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:08 UTC)" (scheduled at 2025-02-07 03:37:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:37:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:37:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 37, 58, 732976), True, datetime.datetime(2025, 2, 7, 3, 37, 58, 732976), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:37:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:37:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:37:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:37:59,236 [235] [DEBUG] [app] Starting request: urn:request:67cc4a85-3618-46c3-b194-e36d112b54c1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:59,236 [238] [DEBUG] [app] Starting request: urn:request:5d1fa86d-e848-42d7-8de2-32c1c211e513 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:37:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:37:59,239 [239] [DEBUG] [app] Starting request: urn:request:e6cf06a8-0536-45cd-a7d3-65a03d7b938a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:59,239 [246] [DEBUG] [app] Starting request: urn:request:3c0f1530-7374-45e4-9d02-3bd8e726c651 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:37:59,240 [239] [DEBUG] [app] Ending request: urn:request:e6cf06a8-0536-45cd-a7d3-65a03d7b938a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e6cf06a8-0536-45cd-a7d3-65a03d7b938a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:37:59,240 [246] [DEBUG] [app] Ending request: urn:request:3c0f1530-7374-45e4-9d02-3bd8e726c651 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3c0f1530-7374-45e4-9d02-3bd8e726c651', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:37:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:37:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:37:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:37:59,243 [238] [DEBUG] [app] Starting request: urn:request:fd4e6263-f8e7-4dbf-8040-2d778714c7e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:59,243 [237] [DEBUG] [app] Starting request: urn:request:1bbc570f-9b4e-415e-9b42-28e15408e6c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:37:59,243 [238] [DEBUG] [app] Ending request: urn:request:fd4e6263-f8e7-4dbf-8040-2d778714c7e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd4e6263-f8e7-4dbf-8040-2d778714c7e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:37:59,243 [237] [DEBUG] [app] Ending request: urn:request:1bbc570f-9b4e-415e-9b42-28e15408e6c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1bbc570f-9b4e-415e-9b42-28e15408e6c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:37:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:37:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:37:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:37:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:37:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:37:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:37:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:37:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:37:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:37:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:37:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:37:59,256 [238] [DEBUG] [app] Ending request: urn:request:5d1fa86d-e848-42d7-8de2-32c1c211e513 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d1fa86d-e848-42d7-8de2-32c1c211e513', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:37:59,256 [235] [DEBUG] [app] Ending request: urn:request:67cc4a85-3618-46c3-b194-e36d112b54c1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:67cc4a85-3618-46c3-b194-e36d112b54c1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:37:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:37:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:37:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:37:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:37:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:37:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 03:37:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:37:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:04.936688+00:00 (in 4.998280 seconds)
exportactionlogsworker stdout | 2025-02-07 03:37:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:59 UTC)" (scheduled at 2025-02-07 03:37:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:37:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:37:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:38:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:38:02,901 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:38:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:38:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:23.420886+00:00 (in 20.001288 seconds)
repositorygcworker stdout | 2025-02-07 03:38:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:03 UTC)" (scheduled at 2025-02-07 03:38:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:38:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:38:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 3, 419881), True, datetime.datetime(2025, 2, 7, 3, 38, 3, 419881), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:38:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:38:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:38:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:38:03,901 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
buildlogsarchiver stdout | 2025-02-07 03:38:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:38:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:33.919286+00:00 (in 29.999521 seconds)
buildlogsarchiver stdout | 2025-02-07 03:38:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:33 UTC)" (scheduled at 2025-02-07 03:38:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:38:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 38, 3, 920041), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:38:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:38:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:38:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:38:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:38:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:59.937899+00:00 (in 55.000755 seconds)
exportactionlogsworker stdout | 2025-02-07 03:38:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:04 UTC)" (scheduled at 2025-02-07 03:38:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:38:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:38:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 4, 937385), True, datetime.datetime(2025, 2, 7, 3, 38, 4, 937385), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:38:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:38:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:38:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:38:08,570 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:38:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:38:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:18.732147+00:00 (in 9.999329 seconds)
notificationworker stdout | 2025-02-07 03:38:08,733 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:18 UTC)" (scheduled at 2025-02-07 03:38:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:38:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:38:08,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 8, 733283), True, datetime.datetime(2025, 2, 7, 3, 38, 8, 733283), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:38:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:38:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:38:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:38:09,099 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:38:10,271 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:38:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:38:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:24.423055+00:00 (in 13.998255 seconds)
gcworker stdout | 2025-02-07 03:38:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:40 UTC)" (scheduled at 2025-02-07 03:38:10.424250+00:00)
gcworker stdout | 2025-02-07 03:38:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:38:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899190431, None, 1, 0])
gcworker stdout | 2025-02-07 03:38:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:38:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:38:11,140 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:38:11,215 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:38:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:38:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:41.831521+00:00 (in 29.999542 seconds)
autopruneworker stdout | 2025-02-07 03:38:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:41 UTC)" (scheduled at 2025-02-07 03:38:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:38:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895891836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:38:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:38:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:38:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:38:12,298 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:12,602 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:38:13,284 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:38:13,386 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:38:13,415 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:38:13,811 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:38:14,235 [238] [DEBUG] [app] Starting request: urn:request:08da3201-cfe1-4401-92b7-4c11caf795e0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:14,235 [235] [DEBUG] [app] Starting request: urn:request:06325469-9846-4df1-bc1c-2fe0aa5f7aa4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:38:14,238 [239] [DEBUG] [app] Starting request: urn:request:228d5d2c-bc58-4989-af68-d6ba90e9a427 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:14,238 [245] [DEBUG] [app] Starting request: urn:request:184ff5a7-0023-4d2c-a01b-a82f32d83ff9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:14,239 [239] [DEBUG] [app] Ending request: urn:request:228d5d2c-bc58-4989-af68-d6ba90e9a427 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:228d5d2c-bc58-4989-af68-d6ba90e9a427', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:14,239 [245] [DEBUG] [app] Ending request: urn:request:184ff5a7-0023-4d2c-a01b-a82f32d83ff9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:184ff5a7-0023-4d2c-a01b-a82f32d83ff9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:38:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:38:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:14,244 [236] [DEBUG] [app] Starting request: urn:request:b1022ed4-6e69-4b4a-b72c-0cf2943d2735 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:38:14,244 [237] [DEBUG] [app] Starting request: urn:request:6085a222-1b34-4d27-a440-53f06b9745d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:38:14,244 [236] [DEBUG] [app] Ending request: urn:request:b1022ed4-6e69-4b4a-b72c-0cf2943d2735 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b1022ed4-6e69-4b4a-b72c-0cf2943d2735', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:38:14,244 [237] [DEBUG] [app] Ending request: urn:request:6085a222-1b34-4d27-a440-53f06b9745d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6085a222-1b34-4d27-a440-53f06b9745d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 159 0.002)
gunicorn-web stdout | 2025-02-07 03:38:14,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:14,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:14,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:14,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:38:14,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:14,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:14,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:38:14,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:38:14,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:14,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:14,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:14,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:14,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:14,255 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:14,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:14,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:14,257 [235] [DEBUG] [app] Ending request: urn:request:06325469-9846-4df1-bc1c-2fe0aa5f7aa4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:06325469-9846-4df1-bc1c-2fe0aa5f7aa4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:14,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:38:14,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 03:38:14,257 [238] [DEBUG] [app] Ending request: urn:request:08da3201-cfe1-4401-92b7-4c11caf795e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08da3201-cfe1-4401-92b7-4c11caf795e0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:14,258 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:38:14,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gcworker stdout | 2025-02-07 03:38:14,311 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:38:14,356 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:38:14,408 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:38:15,910 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:38:15,913 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:38:15,918 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:38:15,921 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:38:15,924 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:38:16,056 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:38:16,418 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:38:16,651 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:38:16,940 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:38:16,943 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:38:16,945 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:38:17,898 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:38:17,905 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:38:17,908 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:38:17,913 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:38:17,918 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:38:17,921 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:38:17,924 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:38:17,927 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:38:17,930 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:38:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:38:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:28.732147+00:00 (in 9.999479 seconds)
notificationworker stdout | 2025-02-07 03:38:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:28 UTC)" (scheduled at 2025-02-07 03:38:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:38:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:38:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 18, 732944), True, datetime.datetime(2025, 2, 7, 3, 38, 18, 732944), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:38:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:38:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:38:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:38:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:38:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:32.420707+00:00 (in 8.999355 seconds)
repositorygcworker stdout | 2025-02-07 03:38:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:23 UTC)" (scheduled at 2025-02-07 03:38:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:38:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:38:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:38:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:38:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:40.424250+00:00 (in 16.000720 seconds)
gcworker stdout | 2025-02-07 03:38:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:54 UTC)" (scheduled at 2025-02-07 03:38:24.423055+00:00)
gcworker stdout | 2025-02-07 03:38:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:38:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:38:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:38:25,628 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:38:27,609 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:38:28,598 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:38:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:38:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:38.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:38:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:38 UTC)" (scheduled at 2025-02-07 03:38:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:38:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:38:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 28, 732862), True, datetime.datetime(2025, 2, 7, 3, 38, 28, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:38:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:38:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:38:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:38:29,235 [237] [DEBUG] [app] Starting request: urn:request:fb94fac9-9934-48e0-a4ac-6abd97cbff41 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:29,235 [236] [DEBUG] [app] Starting request: urn:request:ae806df9-d5fc-4490-904d-d2d035234887 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:38:29,239 [239] [DEBUG] [app] Starting request: urn:request:ed7d4a44-4d14-4655-b571-1a4ac1714e56 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:29,239 [246] [DEBUG] [app] Starting request: urn:request:b4d67fc3-c282-46c5-9b49-7aae2bf0cced (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:29,239 [239] [DEBUG] [app] Ending request: urn:request:ed7d4a44-4d14-4655-b571-1a4ac1714e56 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ed7d4a44-4d14-4655-b571-1a4ac1714e56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:29,239 [246] [DEBUG] [app] Ending request: urn:request:b4d67fc3-c282-46c5-9b49-7aae2bf0cced (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b4d67fc3-c282-46c5-9b49-7aae2bf0cced', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:38:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:29,242 [237] [DEBUG] [app] Starting request: urn:request:8c83c529-5ba6-44e8-92d0-85b5f238234c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:38:29,243 [236] [DEBUG] [app] Starting request: urn:request:de0fb4b8-8c05-4ea3-b444-7f67af528e27 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:38:29,243 [237] [DEBUG] [app] Ending request: urn:request:8c83c529-5ba6-44e8-92d0-85b5f238234c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c83c529-5ba6-44e8-92d0-85b5f238234c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:38:29,243 [236] [DEBUG] [app] Ending request: urn:request:de0fb4b8-8c05-4ea3-b444-7f67af528e27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de0fb4b8-8c05-4ea3-b444-7f67af528e27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:38:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:38:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:38:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:38:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:29,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:29,255 [236] [DEBUG] [app] Ending request: urn:request:ae806df9-d5fc-4490-904d-d2d035234887 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ae806df9-d5fc-4490-904d-d2d035234887', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:38:29,255 [237] [DEBUG] [app] Ending request: urn:request:fb94fac9-9934-48e0-a4ac-6abd97cbff41 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb94fac9-9934-48e0-a4ac-6abd97cbff41', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:38:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:38:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:38:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:38:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:32.625945+00:00 (in 1.004344 seconds)
namespacegcworker stdout | 2025-02-07 03:38:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:31 UTC)" (scheduled at 2025-02-07 03:38:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:38:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:38:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 31, 621904), True, datetime.datetime(2025, 2, 7, 3, 38, 31, 621904), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:38:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:38:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:38:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:38:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:38:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:03.419110+00:00 (in 30.997902 seconds)
repositorygcworker stdout | 2025-02-07 03:38:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:43:32 UTC)" (scheduled at 2025-02-07 03:38:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:38:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 38, 32, 421538), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:38:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 38, 32, 421538), True, datetime.datetime(2025, 2, 7, 3, 38, 32, 421538), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:38:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 38, 32, 421538), True, datetime.datetime(2025, 2, 7, 3, 38, 32, 421538), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 38, 32, 421538), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:38:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:38:32,432 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:43:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:38:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:38:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:31.621131+00:00 (in 58.994720 seconds)
namespacegcworker stdout | 2025-02-07 03:38:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:32 UTC)" (scheduled at 2025-02-07 03:38:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:38:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:38:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:38:32,937 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:38:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:38:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:03.919286+00:00 (in 29.999539 seconds)
buildlogsarchiver stdout | 2025-02-07 03:38:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:03 UTC)" (scheduled at 2025-02-07 03:38:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:38:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 38, 33, 919997), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:38:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:38:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:38:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:38:33,937 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:56.713547+00:00 (in 18.997128 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:37 UTC)" (scheduled at 2025-02-07 03:38:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:38:38,608 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:38:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:38:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:48.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 03:38:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:48 UTC)" (scheduled at 2025-02-07 03:38:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:38:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:38:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 38, 732864), True, datetime.datetime(2025, 2, 7, 3, 38, 38, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:38:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:38:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:38:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:38:39,118 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:38:40,305 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:38:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:38:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:54.423055+00:00 (in 13.998322 seconds)
gcworker stdout | 2025-02-07 03:38:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:10 UTC)" (scheduled at 2025-02-07 03:38:40.424250+00:00)
gcworker stdout | 2025-02-07 03:38:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:38:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899220431, None, 1, 0])
gcworker stdout | 2025-02-07 03:38:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:38:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:38:41,177 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:38:41,251 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:38:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:38:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:11.831521+00:00 (in 29.999547 seconds)
autopruneworker stdout | 2025-02-07 03:38:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:11 UTC)" (scheduled at 2025-02-07 03:38:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:38:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895921836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:38:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:38:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:38:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:38:42,334 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:42,638 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:38:43,293 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:38:43,422 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:38:43,451 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:38:43,843 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:38:44,235 [238] [DEBUG] [app] Starting request: urn:request:ba52f0fd-19a6-4a92-83c3-e93b954decf5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:44,235 [235] [DEBUG] [app] Starting request: urn:request:0c866d36-6a1c-4157-b3b2-5306312854b2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:38:44,238 [239] [DEBUG] [app] Starting request: urn:request:71ee90a6-1168-41c4-b62a-bf255387a3d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:44,238 [245] [DEBUG] [app] Starting request: urn:request:cff7abb0-eb56-47b7-94bb-662a06679baa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:44,238 [239] [DEBUG] [app] Ending request: urn:request:71ee90a6-1168-41c4-b62a-bf255387a3d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71ee90a6-1168-41c4-b62a-bf255387a3d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:44,239 [245] [DEBUG] [app] Ending request: urn:request:cff7abb0-eb56-47b7-94bb-662a06679baa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cff7abb0-eb56-47b7-94bb-662a06679baa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:38:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:38:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:38:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:44,242 [237] [DEBUG] [app] Starting request: urn:request:ed939dec-3642-473c-8469-c280e0f5f292 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:38:44,242 [236] [DEBUG] [app] Starting request: urn:request:de0c5fc8-1112-455c-bd09-43541171a18a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:38:44,243 [237] [DEBUG] [app] Ending request: urn:request:ed939dec-3642-473c-8469-c280e0f5f292 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ed939dec-3642-473c-8469-c280e0f5f292', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:38:44,243 [236] [DEBUG] [app] Ending request: urn:request:de0c5fc8-1112-455c-bd09-43541171a18a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de0c5fc8-1112-455c-bd09-43541171a18a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:38:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:38:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:38:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:38:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:44,255 [238] [DEBUG] [app] Ending request: urn:request:ba52f0fd-19a6-4a92-83c3-e93b954decf5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba52f0fd-19a6-4a92-83c3-e93b954decf5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:44,255 [235] [DEBUG] [app] Ending request: urn:request:0c866d36-6a1c-4157-b3b2-5306312854b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c866d36-6a1c-4157-b3b2-5306312854b2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:38:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:38:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:38:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gcworker stdout | 2025-02-07 03:38:44,343 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:38:44,392 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:38:44,442 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:38:45,919 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:38:45,922 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:38:45,925 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:38:45,929 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:38:45,932 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:38:46,089 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:38:46,454 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:38:46,658 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:38:46,949 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:38:46,951 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:38:46,954 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:38:47,908 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:38:47,916 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:38:47,918 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:38:47,922 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:38:47,926 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:38:47,930 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:38:47,933 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:38:47,936 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:38:47,938 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:38:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:38:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:50.814677+00:00 (in 2.082093 seconds)
notificationworker stdout | 2025-02-07 03:38:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:58 UTC)" (scheduled at 2025-02-07 03:38:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:38:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:38:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 48, 732959), True, datetime.datetime(2025, 2, 7, 3, 38, 48, 732959), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:38:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:38:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:38:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:38:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:38:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:38:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:38:58.732147+00:00 (in 7.916903 seconds)
notificationworker stdout | 2025-02-07 03:38:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:50 UTC)" (scheduled at 2025-02-07 03:38:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:38:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:38:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:38:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:38:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:10.424250+00:00 (in 16.000676 seconds)
gcworker stdout | 2025-02-07 03:38:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:24 UTC)" (scheduled at 2025-02-07 03:38:54.423055+00:00)
gcworker stdout | 2025-02-07 03:38:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:38:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:38:55,655 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:38:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:38:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:56.118855+00:00 (in 59.999466 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:38:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:56 UTC)" (scheduled at 2025-02-07 03:38:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:38:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:38:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:38:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:37.715878+00:00 (in 41.001887 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:56 UTC)" (scheduled at 2025-02-07 03:38:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:38:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:38:57,641 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:38:58,627 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:38:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:38:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:08.732147+00:00 (in 9.999478 seconds)
notificationworker stdout | 2025-02-07 03:38:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:08 UTC)" (scheduled at 2025-02-07 03:38:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:38:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:38:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 38, 58, 732976), True, datetime.datetime(2025, 2, 7, 3, 38, 58, 732976), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:38:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:38:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:38:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:38:59,236 [237] [DEBUG] [app] Starting request: urn:request:6840114a-5565-4c39-85c1-67f126f3c4d9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:59,236 [235] [DEBUG] [app] Starting request: urn:request:898bcb15-369e-4e0c-a173-ecde760aba43 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:38:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:38:59,239 [239] [DEBUG] [app] Starting request: urn:request:5baf70c8-777f-4f39-81ce-2af520cc07a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:59,239 [244] [DEBUG] [app] Starting request: urn:request:1838c6f0-baea-4716-8ad6-dbbc5a98b424 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:38:59,239 [239] [DEBUG] [app] Ending request: urn:request:5baf70c8-777f-4f39-81ce-2af520cc07a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5baf70c8-777f-4f39-81ce-2af520cc07a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:59,239 [244] [DEBUG] [app] Ending request: urn:request:1838c6f0-baea-4716-8ad6-dbbc5a98b424 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1838c6f0-baea-4716-8ad6-dbbc5a98b424', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:38:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:38:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:38:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:38:59,242 [237] [DEBUG] [app] Starting request: urn:request:946d99ba-c91c-432a-9a65-f0aacbb00ce0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:38:59,242 [237] [DEBUG] [app] Ending request: urn:request:946d99ba-c91c-432a-9a65-f0aacbb00ce0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:946d99ba-c91c-432a-9a65-f0aacbb00ce0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:38:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:59,242 [236] [DEBUG] [app] Starting request: urn:request:ac9811f6-8d5f-4aa2-aaa5-aeaf1dd03770 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:38:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:59,243 [236] [DEBUG] [app] Ending request: urn:request:ac9811f6-8d5f-4aa2-aaa5-aeaf1dd03770 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ac9811f6-8d5f-4aa2-aaa5-aeaf1dd03770', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:38:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:38:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:38:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:38:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:38:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:38:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:38:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:38:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:38:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:38:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:38:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:38:59,254 [237] [DEBUG] [app] Ending request: urn:request:6840114a-5565-4c39-85c1-67f126f3c4d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6840114a-5565-4c39-85c1-67f126f3c4d9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:59,254 [235] [DEBUG] [app] Ending request: urn:request:898bcb15-369e-4e0c-a173-ecde760aba43 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:898bcb15-369e-4e0c-a173-ecde760aba43', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:38:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:38:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:38:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:38:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:38:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:38:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 03:38:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:38:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:04.936688+00:00 (in 4.998294 seconds)
exportactionlogsworker stdout | 2025-02-07 03:38:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:59 UTC)" (scheduled at 2025-02-07 03:38:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:38:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:38:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:39:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:39:02,962 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:39:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:39:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:23.420886+00:00 (in 20.001225 seconds)
repositorygcworker stdout | 2025-02-07 03:39:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:03 UTC)" (scheduled at 2025-02-07 03:39:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:39:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:39:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 3, 419939), True, datetime.datetime(2025, 2, 7, 3, 39, 3, 419939), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:39:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:39:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:39:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:39:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:39:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:33.919286+00:00 (in 29.999546 seconds)
buildlogsarchiver stdout | 2025-02-07 03:39:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:33 UTC)" (scheduled at 2025-02-07 03:39:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:39:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 39, 3, 919998), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:39:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:39:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:39:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:39:03,961 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:39:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:39:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:52.937648+00:00 (in 48.000491 seconds)
exportactionlogsworker stdout | 2025-02-07 03:39:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:04 UTC)" (scheduled at 2025-02-07 03:39:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:39:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:39:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 4, 937405), True, datetime.datetime(2025, 2, 7, 3, 39, 4, 937405), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:39:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:39:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:39:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:39:08,628 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:39:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:39:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:18.732147+00:00 (in 9.999476 seconds)
notificationworker stdout | 2025-02-07 03:39:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:18 UTC)" (scheduled at 2025-02-07 03:39:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:39:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:39:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 8, 732979), True, datetime.datetime(2025, 2, 7, 3, 39, 8, 732979), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:39:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:39:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:39:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:39:09,129 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:39:10,330 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:39:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:39:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:24.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 03:39:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:40 UTC)" (scheduled at 2025-02-07 03:39:10.424250+00:00)
gcworker stdout | 2025-02-07 03:39:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:39:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899250430, None, 1, 0])
gcworker stdout | 2025-02-07 03:39:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:39:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:39:11,206 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:39:11,285 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:39:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:39:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 03:39:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:41 UTC)" (scheduled at 2025-02-07 03:39:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:39:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895951836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:39:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:39:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:39:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:39:12,353 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:12,673 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:39:13,318 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:39:13,458 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:39:13,471 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:39:13,851 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:39:14,235 [237] [DEBUG] [app] Starting request: urn:request:9e0b5380-84c2-4e63-ab99-3319d9c14ba7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:14,235 [238] [DEBUG] [app] Starting request: urn:request:904b2b1f-8856-4923-8b4d-783e01dfba02 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:39:14,238 [245] [DEBUG] [app] Starting request: urn:request:f5f5fcac-741e-404c-818c-e3d1ec3bded8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:39:14,239 [239] [DEBUG] [app] Starting request: urn:request:33288d43-774e-437f-afaf-4a42990128bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:39:14,239 [245] [DEBUG] [app] Ending request: urn:request:f5f5fcac-741e-404c-818c-e3d1ec3bded8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5f5fcac-741e-404c-818c-e3d1ec3bded8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:39:14,239 [239] [DEBUG] [app] Ending request: urn:request:33288d43-774e-437f-afaf-4a42990128bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:33288d43-774e-437f-afaf-4a42990128bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:39:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:39:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:39:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:14,242 [238] [DEBUG] [app] Starting request: urn:request:2de498f4-2514-4bd3-a5e3-0ce6950e6a26 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:14,242 [238] [DEBUG] [app] Ending request: urn:request:2de498f4-2514-4bd3-a5e3-0ce6950e6a26 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2de498f4-2514-4bd3-a5e3-0ce6950e6a26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:39:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:39:14,243 [235] [DEBUG] [app] Starting request: urn:request:26ea5819-63a4-477b-8d41-1d760ee1ff91 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:14,243 [235] [DEBUG] [app] Ending request: urn:request:26ea5819-63a4-477b-8d41-1d760ee1ff91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26ea5819-63a4-477b-8d41-1d760ee1ff91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:39:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:14,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:39:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:39:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:14,255 [238] [DEBUG] [app] Ending request: urn:request:904b2b1f-8856-4923-8b4d-783e01dfba02 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:904b2b1f-8856-4923-8b4d-783e01dfba02', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:39:14,256 [237] [DEBUG] [app] Ending request: urn:request:9e0b5380-84c2-4e63-ab99-3319d9c14ba7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e0b5380-84c2-4e63-ab99-3319d9c14ba7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:39:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:39:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gcworker stdout | 2025-02-07 03:39:14,380 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:39:14,428 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:39:14,478 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:39:15,929 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:39:15,932 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:39:15,937 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:39:15,941 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:39:15,944 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:39:16,127 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:39:16,476 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:39:16,668 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:39:16,957 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:39:16,960 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:39:16,963 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:39:17,921 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:39:17,927 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:39:17,930 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:39:17,933 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:39:17,936 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:39:17,938 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:39:17,942 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:39:17,944 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:39:17,947 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:39:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:39:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:28.732147+00:00 (in 9.999515 seconds)
notificationworker stdout | 2025-02-07 03:39:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:28 UTC)" (scheduled at 2025-02-07 03:39:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:39:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:39:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 18, 732924), True, datetime.datetime(2025, 2, 7, 3, 39, 18, 732924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:39:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:39:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:39:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:39:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:39:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:03.419110+00:00 (in 39.997758 seconds)
repositorygcworker stdout | 2025-02-07 03:39:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:23 UTC)" (scheduled at 2025-02-07 03:39:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:39:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:39:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:39:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:39:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:40.424250+00:00 (in 16.000727 seconds)
gcworker stdout | 2025-02-07 03:39:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:54 UTC)" (scheduled at 2025-02-07 03:39:24.423055+00:00)
gcworker stdout | 2025-02-07 03:39:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:39:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:39:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:39:25,691 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:39:27,659 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:39:28,663 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:39:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:39:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:38.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:39:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:38 UTC)" (scheduled at 2025-02-07 03:39:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:39:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:39:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 28, 732866), True, datetime.datetime(2025, 2, 7, 3, 39, 28, 732866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:39:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:39:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:39:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:39:29,235 [238] [DEBUG] [app] Starting request: urn:request:dd73f9e2-ac63-4f6a-b72d-0d7c31e392dd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:29,235 [237] [DEBUG] [app] Starting request: urn:request:65b725ad-e944-4bb0-b6a7-eedf14a398d8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:39:29,238 [239] [DEBUG] [app] Starting request: urn:request:01fb43c2-b574-49e8-84da-67235f9b54b4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:39:29,239 [239] [DEBUG] [app] Ending request: urn:request:01fb43c2-b574-49e8-84da-67235f9b54b4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01fb43c2-b574-49e8-84da-67235f9b54b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:39:29,239 [244] [DEBUG] [app] Starting request: urn:request:7dbb3af2-ccbf-4c07-9a4a-54135f2b152b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:39:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:39:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:39:29,239 [244] [DEBUG] [app] Ending request: urn:request:7dbb3af2-ccbf-4c07-9a4a-54135f2b152b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7dbb3af2-ccbf-4c07-9a4a-54135f2b152b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:39:29,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:39:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:29,242 [238] [DEBUG] [app] Starting request: urn:request:c4badd30-e764-4639-a9d8-0cb4aeeccd77 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:29,242 [238] [DEBUG] [app] Ending request: urn:request:c4badd30-e764-4639-a9d8-0cb4aeeccd77 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4badd30-e764-4639-a9d8-0cb4aeeccd77', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:39:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:39:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:29,243 [235] [DEBUG] [app] Starting request: urn:request:11d64940-7761-43c7-92c0-4d7f4371ec51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:29,243 [235] [DEBUG] [app] Ending request: urn:request:11d64940-7761-43c7-92c0-4d7f4371ec51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11d64940-7761-43c7-92c0-4d7f4371ec51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:39:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:39:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:29,253 [238] [DEBUG] [app] Ending request: urn:request:dd73f9e2-ac63-4f6a-b72d-0d7c31e392dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dd73f9e2-ac63-4f6a-b72d-0d7c31e392dd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:39:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:39:29,254 [237] [DEBUG] [app] Ending request: urn:request:65b725ad-e944-4bb0-b6a7-eedf14a398d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:65b725ad-e944-4bb0-b6a7-eedf14a398d8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:39:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:39:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:39:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:32.625945+00:00 (in 1.004346 seconds)
namespacegcworker stdout | 2025-02-07 03:39:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:31 UTC)" (scheduled at 2025-02-07 03:39:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:39:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:39:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 31, 621832), True, datetime.datetime(2025, 2, 7, 3, 39, 31, 621832), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:39:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:39:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:39:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:39:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:39:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:31.621131+00:00 (in 58.994718 seconds)
namespacegcworker stdout | 2025-02-07 03:39:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:32 UTC)" (scheduled at 2025-02-07 03:39:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:39:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:39:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:39:32,998 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:39:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:39:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:03.919286+00:00 (in 29.999519 seconds)
buildlogsarchiver stdout | 2025-02-07 03:39:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:03 UTC)" (scheduled at 2025-02-07 03:39:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:39:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 39, 33, 920031), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:39:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:39:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:39:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:39:33,998 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:56.713547+00:00 (in 18.997198 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:37 UTC)" (scheduled at 2025-02-07 03:39:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:39:38,640 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:39:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:39:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:48.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:39:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:48 UTC)" (scheduled at 2025-02-07 03:39:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:39:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:39:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 38, 732866), True, datetime.datetime(2025, 2, 7, 3, 39, 38, 732866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:39:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:39:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:39:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:39:39,165 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:39:40,343 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:39:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:39:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:54.423055+00:00 (in 13.998370 seconds)
gcworker stdout | 2025-02-07 03:39:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:10 UTC)" (scheduled at 2025-02-07 03:39:40.424250+00:00)
gcworker stdout | 2025-02-07 03:39:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:39:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899280431, None, 1, 0])
gcworker stdout | 2025-02-07 03:39:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:39:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:39:41,231 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:39:41,292 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:39:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:39:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:11.831521+00:00 (in 29.999564 seconds)
autopruneworker stdout | 2025-02-07 03:39:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:11 UTC)" (scheduled at 2025-02-07 03:39:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:39:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738895981836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:39:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:39:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:39:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:39:42,389 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:42,706 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:39:43,343 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:39:43,495 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:39:43,498 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:39:43,871 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:39:44,236 [235] [DEBUG] [app] Starting request: urn:request:466e3fdd-62f4-448b-9331-3c866fb99d25 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:44,236 [238] [DEBUG] [app] Starting request: urn:request:d238dc3c-ff9d-4f59-ab99-1cb9592c400a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:39:44,239 [246] [DEBUG] [app] Starting request: urn:request:7b7bd805-a3c0-4e24-9368-d4ddb6d10324 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:39:44,239 [245] [DEBUG] [app] Starting request: urn:request:01645964-2b54-4637-aadc-d0ec21735ed6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:39:44,239 [246] [DEBUG] [app] Ending request: urn:request:7b7bd805-a3c0-4e24-9368-d4ddb6d10324 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b7bd805-a3c0-4e24-9368-d4ddb6d10324', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:39:44,239 [245] [DEBUG] [app] Ending request: urn:request:01645964-2b54-4637-aadc-d0ec21735ed6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01645964-2b54-4637-aadc-d0ec21735ed6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:39:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:39:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:39:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:44,242 [238] [DEBUG] [app] Starting request: urn:request:10df35f2-8b8d-4fca-b3a1-52bc92415fc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:44,242 [238] [DEBUG] [app] Ending request: urn:request:10df35f2-8b8d-4fca-b3a1-52bc92415fc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10df35f2-8b8d-4fca-b3a1-52bc92415fc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:39:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:39:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:44,243 [237] [DEBUG] [app] Starting request: urn:request:6060b2af-15c3-46e0-a7ff-990bbf86b34b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:44,243 [237] [DEBUG] [app] Ending request: urn:request:6060b2af-15c3-46e0-a7ff-990bbf86b34b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6060b2af-15c3-46e0-a7ff-990bbf86b34b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:39:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:39:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:44,254 [238] [DEBUG] [app] Ending request: urn:request:d238dc3c-ff9d-4f59-ab99-1cb9592c400a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d238dc3c-ff9d-4f59-ab99-1cb9592c400a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:39:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:39:44,255 [235] [DEBUG] [app] Ending request: urn:request:466e3fdd-62f4-448b-9331-3c866fb99d25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:466e3fdd-62f4-448b-9331-3c866fb99d25', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:39:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gcworker stdout | 2025-02-07 03:39:44,417 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:39:44,449 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:39:44,514 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:39:45,936 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:39:45,940 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:39:45,945 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:39:45,948 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:39:45,951 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:39:46,141 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:39:46,512 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:39:46,682 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:39:46,966 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:39:46,969 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:39:46,971 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:39:47,931 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:39:47,936 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:39:47,939 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:39:47,942 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:39:47,947 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:39:47,951 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:39:47,954 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:39:47,957 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:39:47,959 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:39:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:39:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:50.814677+00:00 (in 2.082091 seconds)
notificationworker stdout | 2025-02-07 03:39:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:58 UTC)" (scheduled at 2025-02-07 03:39:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:39:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:39:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 48, 732874), True, datetime.datetime(2025, 2, 7, 3, 39, 48, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:39:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:39:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:39:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:39:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:39:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:39:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:58.732147+00:00 (in 7.917005 seconds)
notificationworker stdout | 2025-02-07 03:39:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:50 UTC)" (scheduled at 2025-02-07 03:39:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:39:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:39:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:39:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:39:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:39:59.937899+00:00 (in 6.999720 seconds)
exportactionlogsworker stdout | 2025-02-07 03:39:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:44:52 UTC)" (scheduled at 2025-02-07 03:39:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:39:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 39, 52, 938497), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:39:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 39, 52, 938497), True, datetime.datetime(2025, 2, 7, 3, 39, 52, 938497), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:39:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 39, 52, 938497), True, datetime.datetime(2025, 2, 7, 3, 39, 52, 938497), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 39, 52, 938497), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:39:52,949 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:39:52,949 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:44:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:39:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:39:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:10.424250+00:00 (in 16.000702 seconds)
gcworker stdout | 2025-02-07 03:39:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:24 UTC)" (scheduled at 2025-02-07 03:39:54.423055+00:00)
gcworker stdout | 2025-02-07 03:39:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:39:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:39:55,728 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:39:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:39:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:56.118855+00:00 (in 59.999488 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:39:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:56 UTC)" (scheduled at 2025-02-07 03:39:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:39:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:39:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:39:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:37.715878+00:00 (in 41.001868 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:56 UTC)" (scheduled at 2025-02-07 03:39:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:56,715 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:39:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:39:57,689 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 03:39:58,699 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:39:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:39:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:08.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:39:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:08 UTC)" (scheduled at 2025-02-07 03:39:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:39:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:39:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 39, 58, 732915), True, datetime.datetime(2025, 2, 7, 3, 39, 58, 732915), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:39:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:39:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:39:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:08 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:39:59,236 [235] [DEBUG] [app] Starting request: urn:request:4dd6f889-4fb2-4de2-8b6f-820db19a7b2a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:59,236 [238] [DEBUG] [app] Starting request: urn:request:8151f8ab-c539-4bac-afaf-4473d3de0f01 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:39:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:39:59,238 [246] [DEBUG] [app] Starting request: urn:request:17759edb-a601-4d10-9ba5-0ac227fe8926 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:39:59,238 [239] [DEBUG] [app] Starting request: urn:request:6afd4f89-fd75-42b8-94e0-193c4334426a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:39:59,239 [239] [DEBUG] [app] Ending request: urn:request:6afd4f89-fd75-42b8-94e0-193c4334426a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6afd4f89-fd75-42b8-94e0-193c4334426a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:39:59,239 [246] [DEBUG] [app] Ending request: urn:request:17759edb-a601-4d10-9ba5-0ac227fe8926 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:17759edb-a601-4d10-9ba5-0ac227fe8926', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:39:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:39:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:39:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:39:59,242 [237] [DEBUG] [app] Starting request: urn:request:38b08916-e40f-4112-9a75-38e11dd149c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:59,242 [237] [DEBUG] [app] Ending request: urn:request:38b08916-e40f-4112-9a75-38e11dd149c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:38b08916-e40f-4112-9a75-38e11dd149c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:39:59,242 [235] [DEBUG] [app] Starting request: urn:request:9cb49c79-e930-44a2-bcc2-0a61e7c41fad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:39:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:39:59,243 [235] [DEBUG] [app] Ending request: urn:request:9cb49c79-e930-44a2-bcc2-0a61e7c41fad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9cb49c79-e930-44a2-bcc2-0a61e7c41fad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:39:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:39:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:39:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:39:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:39:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:39:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:39:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:39:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:39:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:39:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:39:59,255 [235] [DEBUG] [app] Ending request: urn:request:4dd6f889-4fb2-4de2-8b6f-820db19a7b2a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4dd6f889-4fb2-4de2-8b6f-820db19a7b2a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:59,255 [238] [DEBUG] [app] Ending request: urn:request:8151f8ab-c539-4bac-afaf-4473d3de0f01 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8151f8ab-c539-4bac-afaf-4473d3de0f01', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:39:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:39:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:39:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:39:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:39:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:39:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:39:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:39:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:04.936688+00:00 (in 4.998247 seconds)
exportactionlogsworker stdout | 2025-02-07 03:39:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:59 UTC)" (scheduled at 2025-02-07 03:39:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:39:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:39:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:40:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:40:03,035 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:40:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:40:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:23.420886+00:00 (in 20.001314 seconds)
repositorygcworker stdout | 2025-02-07 03:40:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:03 UTC)" (scheduled at 2025-02-07 03:40:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:40:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:40:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 3, 419860), True, datetime.datetime(2025, 2, 7, 3, 40, 3, 419860), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:40:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:40:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:40:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:40:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:40:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:33.919286+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2025-02-07 03:40:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:33 UTC)" (scheduled at 2025-02-07 03:40:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:40:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 40, 3, 920029), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:40:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:40:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:40:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:40:04,012 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:40:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:40:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:59.937899+00:00 (in 55.000755 seconds)
exportactionlogsworker stdout | 2025-02-07 03:40:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:04 UTC)" (scheduled at 2025-02-07 03:40:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:40:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:40:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 4, 937387), True, datetime.datetime(2025, 2, 7, 3, 40, 4, 937387), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:40:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:40:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:40:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:40:08,676 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:40:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:40:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:18.732147+00:00 (in 9.999540 seconds)
notificationworker stdout | 2025-02-07 03:40:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:18 UTC)" (scheduled at 2025-02-07 03:40:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:40:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:40:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 8, 732892), True, datetime.datetime(2025, 2, 7, 3, 40, 8, 732892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:40:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:40:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:40:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:40:09,178 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:40:10,353 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:40:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:40:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:24.423055+00:00 (in 13.998364 seconds)
gcworker stdout | 2025-02-07 03:40:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:40 UTC)" (scheduled at 2025-02-07 03:40:10.424250+00:00)
gcworker stdout | 2025-02-07 03:40:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:40:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899310431, None, 1, 0])
gcworker stdout | 2025-02-07 03:40:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:40:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:40:11,267 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:40:11,329 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:40:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:40:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:41.831521+00:00 (in 29.999573 seconds)
autopruneworker stdout | 2025-02-07 03:40:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:41 UTC)" (scheduled at 2025-02-07 03:40:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:40:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896011836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:40:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:40:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:40:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:40:12,425 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:12,732 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:40:13,379 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:40:13,520 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:40:13,529 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:40:13,878 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:40:14,236 [236] [DEBUG] [app] Starting request: urn:request:97427445-11d3-4dc1-b496-e8cb343f742c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:14,236 [237] [DEBUG] [app] Starting request: urn:request:80aaa1fd-9061-4fce-9b23-7f9c0cc4afad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:40:14,239 [246] [DEBUG] [app] Starting request: urn:request:9b1ff836-5131-4199-be98-86646ae3ccd6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:14,239 [244] [DEBUG] [app] Starting request: urn:request:a23b2b1c-08c5-4926-a2a7-4de6e97064e5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:14,240 [244] [DEBUG] [app] Ending request: urn:request:a23b2b1c-08c5-4926-a2a7-4de6e97064e5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a23b2b1c-08c5-4926-a2a7-4de6e97064e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:40:14,240 [246] [DEBUG] [app] Ending request: urn:request:9b1ff836-5131-4199-be98-86646ae3ccd6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b1ff836-5131-4199-be98-86646ae3ccd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:40:14,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:40:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:40:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:14,243 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:14,243 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:14,245 [235] [DEBUG] [app] Starting request: urn:request:314edd49-eb7d-4d22-9cfd-ccd34531009c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:14,245 [238] [DEBUG] [app] Starting request: urn:request:f8c3d512-698e-467a-8f3c-3b8600c7341f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:14,246 [235] [DEBUG] [app] Ending request: urn:request:314edd49-eb7d-4d22-9cfd-ccd34531009c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:314edd49-eb7d-4d22-9cfd-ccd34531009c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:40:14,246 [238] [DEBUG] [app] Ending request: urn:request:f8c3d512-698e-467a-8f3c-3b8600c7341f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f8c3d512-698e-467a-8f3c-3b8600c7341f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:40:14,247 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:40:14,247 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:40:14,247 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:14,247 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:14,247 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:14,247 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:14,248 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:14,248 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:14,252 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:14,252 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:14,257 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:14,257 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:14,259 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:14,259 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:14,260 [237] [DEBUG] [app] Ending request: urn:request:80aaa1fd-9061-4fce-9b23-7f9c0cc4afad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:80aaa1fd-9061-4fce-9b23-7f9c0cc4afad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:14,260 [236] [DEBUG] [app] Ending request: urn:request:97427445-11d3-4dc1-b496-e8cb343f742c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97427445-11d3-4dc1-b496-e8cb343f742c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:14,260 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:40:14,260 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.026 118 0.026)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.026 118 0.026)
gunicorn-web stdout | 2025-02-07 03:40:14,261 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:40:14,261 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gcworker stdout | 2025-02-07 03:40:14,443 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:40:14,485 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:40:14,553 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:40:15,944 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:40:15,946 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:40:15,952 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:40:15,955 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:40:15,958 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:40:16,172 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:40:16,528 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:40:16,690 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:40:16,974 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:40:16,978 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:40:16,980 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:40:17,941 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:40:17,945 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:40:17,948 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:40:17,951 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:40:17,954 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:40:17,959 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:40:17,962 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:40:17,965 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:40:17,970 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:40:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:40:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:28.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 03:40:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:28 UTC)" (scheduled at 2025-02-07 03:40:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:40:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:40:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 18, 732870), True, datetime.datetime(2025, 2, 7, 3, 40, 18, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:40:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:40:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:40:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:40:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:40:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:03.419110+00:00 (in 39.997725 seconds)
repositorygcworker stdout | 2025-02-07 03:40:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:23 UTC)" (scheduled at 2025-02-07 03:40:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:40:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:40:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:40:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:40:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 03:40:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:54 UTC)" (scheduled at 2025-02-07 03:40:24.423055+00:00)
gcworker stdout | 2025-02-07 03:40:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:40:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:40:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:40:25,764 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:40:27,718 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:40:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:40:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:38.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 03:40:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:38 UTC)" (scheduled at 2025-02-07 03:40:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:40:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:40:28,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 28, 732907), True, datetime.datetime(2025, 2, 7, 3, 40, 28, 732907), 0, 'notification/%', 50, 1, 0])
queuecleanupworker stdout | 2025-02-07 03:40:28,737 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 03:40:28,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:40:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:40:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:38 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:40:29,235 [235] [DEBUG] [app] Starting request: urn:request:79cf4c38-e1c1-46d2-9e15-b4e9d8976b8e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:29,235 [236] [DEBUG] [app] Starting request: urn:request:ac7543cd-2430-4e94-9d79-ece7cc20d4fa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:40:29,239 [245] [DEBUG] [app] Starting request: urn:request:6bc18f39-639e-4c80-9a40-4933426fb4f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:29,239 [239] [DEBUG] [app] Starting request: urn:request:0f633a81-d91e-454a-975d-8b7aa2ea56b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:29,240 [239] [DEBUG] [app] Ending request: urn:request:0f633a81-d91e-454a-975d-8b7aa2ea56b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f633a81-d91e-454a-975d-8b7aa2ea56b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:40:29,240 [245] [DEBUG] [app] Ending request: urn:request:6bc18f39-639e-4c80-9a40-4933426fb4f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6bc18f39-639e-4c80-9a40-4933426fb4f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:40:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:40:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:40:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:29,243 [236] [DEBUG] [app] Starting request: urn:request:b2a246ad-4160-4f71-ab58-0d593c22328b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:29,243 [238] [DEBUG] [app] Starting request: urn:request:16be8f35-aedc-4a4e-a262-48f1d36cd45e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:29,243 [236] [DEBUG] [app] Ending request: urn:request:b2a246ad-4160-4f71-ab58-0d593c22328b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2a246ad-4160-4f71-ab58-0d593c22328b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:40:29,243 [238] [DEBUG] [app] Ending request: urn:request:16be8f35-aedc-4a4e-a262-48f1d36cd45e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16be8f35-aedc-4a4e-a262-48f1d36cd45e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:40:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:40:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:29,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:40:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:40:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:29,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:29,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:29,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:29,256 [235] [DEBUG] [app] Ending request: urn:request:79cf4c38-e1c1-46d2-9e15-b4e9d8976b8e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79cf4c38-e1c1-46d2-9e15-b4e9d8976b8e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:29,256 [236] [DEBUG] [app] Ending request: urn:request:ac7543cd-2430-4e94-9d79-ece7cc20d4fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac7543cd-2430-4e94-9d79-ece7cc20d4fa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:29,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:40:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:40:29,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:40:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:40:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:40:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:32.625945+00:00 (in 1.004311 seconds)
namespacegcworker stdout | 2025-02-07 03:40:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:31 UTC)" (scheduled at 2025-02-07 03:40:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:40:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:40:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 31, 621852), True, datetime.datetime(2025, 2, 7, 3, 40, 31, 621852), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:40:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:40:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:40:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:40:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:40:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:31.621131+00:00 (in 58.994695 seconds)
namespacegcworker stdout | 2025-02-07 03:40:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:32 UTC)" (scheduled at 2025-02-07 03:40:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:40:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:40:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:40:33,071 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:40:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:40:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:03.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 03:40:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:03 UTC)" (scheduled at 2025-02-07 03:40:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:40:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 40, 33, 920012), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:40:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:40:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:40:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:40:34,037 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:56.713547+00:00 (in 18.997203 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:37 UTC)" (scheduled at 2025-02-07 03:40:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:37 UTC)" executed successfully
builder stdout | 2025-02-07 03:40:38,712 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:40:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:40:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:48.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 03:40:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:48 UTC)" (scheduled at 2025-02-07 03:40:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:40:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:40:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 38, 732878), True, datetime.datetime(2025, 2, 7, 3, 40, 38, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:40:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:40:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:40:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:48 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:40:39,215 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 03:40:40,389 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:40:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:40:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:54.423055+00:00 (in 13.998367 seconds)
gcworker stdout | 2025-02-07 03:40:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:10 UTC)" (scheduled at 2025-02-07 03:40:40.424250+00:00)
gcworker stdout | 2025-02-07 03:40:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:40:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899340430, None, 1, 0])
gcworker stdout | 2025-02-07 03:40:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:40:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:10 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:40:41,274 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:40:41,365 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:40:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:40:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:11.831521+00:00 (in 29.999548 seconds)
autopruneworker stdout | 2025-02-07 03:40:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:11 UTC)" (scheduled at 2025-02-07 03:40:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:40:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896041836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:40:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:40:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:40:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:40:42,434 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:42,739 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:40:43,415 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:40:43,530 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:40:43,561 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:40:43,907 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:40:44,235 [237] [DEBUG] [app] Starting request: urn:request:a133b2c6-f213-4d3e-bc6b-dd6f44678907 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:44,235 [236] [DEBUG] [app] Starting request: urn:request:8eed8209-370b-4d0d-ae3b-09111f38217e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:40:44,238 [246] [DEBUG] [app] Starting request: urn:request:4361dc42-2a72-4b9e-805e-f8664719624f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:44,238 [245] [DEBUG] [app] Starting request: urn:request:3813dd65-0c9a-4e4f-91d8-6c33b51a8e0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:44,239 [246] [DEBUG] [app] Ending request: urn:request:4361dc42-2a72-4b9e-805e-f8664719624f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4361dc42-2a72-4b9e-805e-f8664719624f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:40:44,239 [245] [DEBUG] [app] Ending request: urn:request:3813dd65-0c9a-4e4f-91d8-6c33b51a8e0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3813dd65-0c9a-4e4f-91d8-6c33b51a8e0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:40:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:40:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:40:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:44,242 [235] [DEBUG] [app] Starting request: urn:request:5eb14557-e143-490f-ad83-8fc4cc49c183 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:44,243 [237] [DEBUG] [app] Starting request: urn:request:34805356-21c4-41c6-a3a1-afaf46fb30ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:44,243 [237] [DEBUG] [app] Ending request: urn:request:34805356-21c4-41c6-a3a1-afaf46fb30ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:34805356-21c4-41c6-a3a1-afaf46fb30ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:40:44,243 [235] [DEBUG] [app] Ending request: urn:request:5eb14557-e143-490f-ad83-8fc4cc49c183 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5eb14557-e143-490f-ad83-8fc4cc49c183', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:40:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:40:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:40:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:44,255 [236] [DEBUG] [app] Ending request: urn:request:8eed8209-370b-4d0d-ae3b-09111f38217e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8eed8209-370b-4d0d-ae3b-09111f38217e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:44,255 [237] [DEBUG] [app] Ending request: urn:request:a133b2c6-f213-4d3e-bc6b-dd6f44678907 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a133b2c6-f213-4d3e-bc6b-dd6f44678907', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:40:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:40:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:40:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gcworker stdout | 2025-02-07 03:40:44,462 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:40:44,521 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:40:44,589 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:40:45,952 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:40:45,955 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:40:45,960 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:40:45,963 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:40:45,966 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:40:46,209 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:40:46,565 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:40:46,718 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:40:46,982 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:40:46,985 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:40:46,987 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:40:47,950 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:40:47,952 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:40:47,955 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:40:47,957 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:40:47,967 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:40:47,969 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:40:47,972 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:40:47,974 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:40:47,979 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:40:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:40:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:50.814677+00:00 (in 2.082090 seconds)
notificationworker stdout | 2025-02-07 03:40:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:58 UTC)" (scheduled at 2025-02-07 03:40:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:40:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:40:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 48, 732869), True, datetime.datetime(2025, 2, 7, 3, 40, 48, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:40:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:40:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:40:48,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:40:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:40:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:40:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:40:58.732147+00:00 (in 7.917005 seconds)
notificationworker stdout | 2025-02-07 03:40:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:50 UTC)" (scheduled at 2025-02-07 03:40:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:40:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:40:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:40:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:40:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:10.424250+00:00 (in 16.000740 seconds)
gcworker stdout | 2025-02-07 03:40:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:24 UTC)" (scheduled at 2025-02-07 03:40:54.423055+00:00)
gcworker stdout | 2025-02-07 03:40:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:40:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:40:55,800 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:40:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:40:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:56.118855+00:00 (in 59.999496 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:40:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:56 UTC)" (scheduled at 2025-02-07 03:40:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:40:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:40:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:40:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:37.715878+00:00 (in 41.001862 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:56 UTC)" (scheduled at 2025-02-07 03:40:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:40:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:40:57,754 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:40:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:40:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:08.732147+00:00 (in 9.999516 seconds)
notificationworker stdout | 2025-02-07 03:40:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:08 UTC)" (scheduled at 2025-02-07 03:40:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:40:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:40:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 40, 58, 732917), True, datetime.datetime(2025, 2, 7, 3, 40, 58, 732917), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:40:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:40:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:40:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:40:58,773 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:40:59,236 [237] [DEBUG] [app] Starting request: urn:request:c5f08e64-aa0b-42e9-9c2e-cdc2f6357974 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:59,236 [235] [DEBUG] [app] Starting request: urn:request:633784ed-8e88-42c2-aab6-2df00a7dba76 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:40:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:59,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:40:59,240 [239] [DEBUG] [app] Starting request: urn:request:aa1c2ad0-e1a8-4a2a-b4ea-707c0dd23511 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:59,240 [244] [DEBUG] [app] Starting request: urn:request:51dd2769-96a0-4cc0-8ef8-40e4001d2df8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:40:59,241 [244] [DEBUG] [app] Ending request: urn:request:51dd2769-96a0-4cc0-8ef8-40e4001d2df8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51dd2769-96a0-4cc0-8ef8-40e4001d2df8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:40:59,241 [239] [DEBUG] [app] Ending request: urn:request:aa1c2ad0-e1a8-4a2a-b4ea-707c0dd23511 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa1c2ad0-e1a8-4a2a-b4ea-707c0dd23511', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:40:59,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:40:59,241 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:40:59,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:59,243 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:59,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:40:59,245 [238] [DEBUG] [app] Starting request: urn:request:b8224634-6463-4fec-8a42-c0149ff48dc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:59,245 [236] [DEBUG] [app] Starting request: urn:request:2847eac8-3a63-4854-9c55-ba6c61bc337a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:40:59,245 [236] [DEBUG] [app] Ending request: urn:request:2847eac8-3a63-4854-9c55-ba6c61bc337a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2847eac8-3a63-4854-9c55-ba6c61bc337a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:40:59,246 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:40:59,246 [238] [DEBUG] [app] Ending request: urn:request:b8224634-6463-4fec-8a42-c0149ff48dc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8224634-6463-4fec-8a42-c0149ff48dc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:40:59,246 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:59,246 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:40:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:40:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:40:59,246 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:40:59,246 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:59,247 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:59,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:40:59,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:40:59,251 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:59,251 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:40:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:59,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:40:59,255 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:59,256 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:40:59,257 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:59,257 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:40:59,258 [235] [DEBUG] [app] Ending request: urn:request:633784ed-8e88-42c2-aab6-2df00a7dba76 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:633784ed-8e88-42c2-aab6-2df00a7dba76', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:59,258 [237] [DEBUG] [app] Ending request: urn:request:c5f08e64-aa0b-42e9-9c2e-cdc2f6357974 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c5f08e64-aa0b-42e9-9c2e-cdc2f6357974', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:40:59,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:40:59,259 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:40:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gunicorn-web stdout | 2025-02-07 03:40:59,259 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:40:59,259 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:40:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:40:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:40:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:04.936688+00:00 (in 4.998292 seconds)
exportactionlogsworker stdout | 2025-02-07 03:40:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:59 UTC)" (scheduled at 2025-02-07 03:40:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:40:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:40:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:41:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:41:03,107 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:41:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:41:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:23.420886+00:00 (in 20.001346 seconds)
repositorygcworker stdout | 2025-02-07 03:41:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:03 UTC)" (scheduled at 2025-02-07 03:41:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:41:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:41:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 3, 419815), True, datetime.datetime(2025, 2, 7, 3, 41, 3, 419815), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:41:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:41:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:41:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:41:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:41:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:33.919286+00:00 (in 29.999565 seconds)
buildlogsarchiver stdout | 2025-02-07 03:41:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:33 UTC)" (scheduled at 2025-02-07 03:41:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:41:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 41, 3, 919969), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:41:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:41:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:41:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:41:04,054 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:41:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:41:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:59.937899+00:00 (in 55.000763 seconds)
exportactionlogsworker stdout | 2025-02-07 03:41:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:04 UTC)" (scheduled at 2025-02-07 03:41:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:41:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:41:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 4, 937385), True, datetime.datetime(2025, 2, 7, 3, 41, 4, 937385), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:41:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:41:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:41:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:04 UTC)" executed successfully
builder stdout | 2025-02-07 03:41:08,731 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
notificationworker stdout | 2025-02-07 03:41:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:41:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:18.732147+00:00 (in 9.999498 seconds)
notificationworker stdout | 2025-02-07 03:41:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:18 UTC)" (scheduled at 2025-02-07 03:41:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:41:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:41:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 8, 732989), True, datetime.datetime(2025, 2, 7, 3, 41, 8, 732989), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:41:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:41:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:41:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:18 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 03:41:09,236 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:41:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:41:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:24.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 03:41:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:40 UTC)" (scheduled at 2025-02-07 03:41:10.424250+00:00)
gcworker stdout | 2025-02-07 03:41:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
expiredappspecifictokenworker stdout | 2025-02-07 03:41:10,426 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gcworker stdout | 2025-02-07 03:41:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899370431, None, 1, 0])
gcworker stdout | 2025-02-07 03:41:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:41:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:40 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:41:11,309 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:41:11,401 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:41:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:41:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 03:41:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:41 UTC)" (scheduled at 2025-02-07 03:41:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:41:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896071836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:41:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:41:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:41:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:41:12,454 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:12,774 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:41:13,446 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:41:13,566 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:41:13,597 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:41:13,935 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:41:14,236 [238] [DEBUG] [app] Starting request: urn:request:41479017-db88-4f57-a571-b592059d327d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:14,236 [235] [DEBUG] [app] Starting request: urn:request:c4bb4683-b7d9-4545-8072-d674052d75fa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:41:14,239 [246] [DEBUG] [app] Starting request: urn:request:ec3e1cdc-19c4-4b32-8d1f-0c69a2c8c10b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:14,239 [245] [DEBUG] [app] Starting request: urn:request:a99c5cd6-f8d0-42bc-a704-0aab50b3df9c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:14,239 [245] [DEBUG] [app] Ending request: urn:request:a99c5cd6-f8d0-42bc-a704-0aab50b3df9c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a99c5cd6-f8d0-42bc-a704-0aab50b3df9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:41:14,239 [246] [DEBUG] [app] Ending request: urn:request:ec3e1cdc-19c4-4b32-8d1f-0c69a2c8c10b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec3e1cdc-19c4-4b32-8d1f-0c69a2c8c10b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:41:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:41:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:41:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:41:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:14,242 [237] [DEBUG] [app] Starting request: urn:request:f94b2d37-2405-405e-8a78-365cdc3e2f10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:14,243 [237] [DEBUG] [app] Ending request: urn:request:f94b2d37-2405-405e-8a78-365cdc3e2f10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f94b2d37-2405-405e-8a78-365cdc3e2f10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:41:14,243 [235] [DEBUG] [app] Starting request: urn:request:9b39ac75-f4a2-46e9-b4a0-97978b91500f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:41:14,243 [235] [DEBUG] [app] Ending request: urn:request:9b39ac75-f4a2-46e9-b4a0-97978b91500f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b39ac75-f4a2-46e9-b4a0-97978b91500f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:41:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:41:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:41:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:41:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:14,255 [235] [DEBUG] [app] Ending request: urn:request:c4bb4683-b7d9-4545-8072-d674052d75fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4bb4683-b7d9-4545-8072-d674052d75fa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:14,255 [238] [DEBUG] [app] Ending request: urn:request:41479017-db88-4f57-a571-b592059d327d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:41479017-db88-4f57-a571-b592059d327d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:41:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:41:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:41:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gcworker stdout | 2025-02-07 03:41:14,475 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:41:14,557 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:41:14,618 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:41:15,959 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:41:15,962 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:41:15,966 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:41:15,970 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:41:15,973 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:41:16,237 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:41:16,602 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:41:16,754 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:41:16,989 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:41:16,992 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:41:16,995 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:41:17,957 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:41:17,960 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:41:17,967 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:41:17,970 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:41:17,976 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:41:17,981 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:41:17,985 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:41:17,987 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:41:17,990 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:41:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:41:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:28.732147+00:00 (in 9.999481 seconds)
notificationworker stdout | 2025-02-07 03:41:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:28 UTC)" (scheduled at 2025-02-07 03:41:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:41:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:41:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 18, 732981), True, datetime.datetime(2025, 2, 7, 3, 41, 18, 732981), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:41:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:41:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:41:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:41:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:41:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:03.419110+00:00 (in 39.997759 seconds)
repositorygcworker stdout | 2025-02-07 03:41:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:23 UTC)" (scheduled at 2025-02-07 03:41:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:41:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:41:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:41:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:41:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:40.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 03:41:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:54 UTC)" (scheduled at 2025-02-07 03:41:24.423055+00:00)
gcworker stdout | 2025-02-07 03:41:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:41:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:41:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:41:25,818 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:41:27,791 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:41:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:41:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:38.732147+00:00 (in 9.999487 seconds)
notificationworker stdout | 2025-02-07 03:41:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:38 UTC)" (scheduled at 2025-02-07 03:41:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:41:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:41:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 28, 732975), True, datetime.datetime(2025, 2, 7, 3, 41, 28, 732975), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:41:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:41:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:41:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:41:28,795 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:41:29,236 [238] [DEBUG] [app] Starting request: urn:request:2cbbe7c8-4d20-4721-8bdc-bd007382b5e5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:29,236 [235] [DEBUG] [app] Starting request: urn:request:d93c6077-bd31-43d3-9739-c4cdd3cbe9de (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:41:29,239 [239] [DEBUG] [app] Starting request: urn:request:15e523e6-b0e9-422b-a1c3-0083bfbef48b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:29,239 [246] [DEBUG] [app] Starting request: urn:request:e7922727-6018-4d68-a428-89803b933e6b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:29,239 [239] [DEBUG] [app] Ending request: urn:request:15e523e6-b0e9-422b-a1c3-0083bfbef48b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15e523e6-b0e9-422b-a1c3-0083bfbef48b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:41:29,240 [246] [DEBUG] [app] Ending request: urn:request:e7922727-6018-4d68-a428-89803b933e6b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e7922727-6018-4d68-a428-89803b933e6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:41:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:41:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:41:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:41:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:29,242 [238] [DEBUG] [app] Starting request: urn:request:aa21b963-76ff-4a59-9254-4fee6efd5290 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:29,243 [238] [DEBUG] [app] Ending request: urn:request:aa21b963-76ff-4a59-9254-4fee6efd5290 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aa21b963-76ff-4a59-9254-4fee6efd5290', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:41:29,243 [237] [DEBUG] [app] Starting request: urn:request:184f4106-fe57-4021-82e7-a60bf1e118d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:41:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:29,243 [237] [DEBUG] [app] Ending request: urn:request:184f4106-fe57-4021-82e7-a60bf1e118d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:184f4106-fe57-4021-82e7-a60bf1e118d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:41:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:41:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:41:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:41:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:41:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:29,255 [235] [DEBUG] [app] Ending request: urn:request:d93c6077-bd31-43d3-9739-c4cdd3cbe9de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d93c6077-bd31-43d3-9739-c4cdd3cbe9de', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:29,255 [238] [DEBUG] [app] Ending request: urn:request:2cbbe7c8-4d20-4721-8bdc-bd007382b5e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2cbbe7c8-4d20-4721-8bdc-bd007382b5e5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:41:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:41:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:41:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:41:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:41:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:32.625945+00:00 (in 1.004344 seconds)
namespacegcworker stdout | 2025-02-07 03:41:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:31 UTC)" (scheduled at 2025-02-07 03:41:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:41:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:41:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 31, 621813), True, datetime.datetime(2025, 2, 7, 3, 41, 31, 621813), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:41:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:41:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:41:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:41:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:41:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:13.624301+00:00 (in 40.997861 seconds)
namespacegcworker stdout | 2025-02-07 03:41:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:32 UTC)" (scheduled at 2025-02-07 03:41:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:41:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:41:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:41:33,118 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:41:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:41:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:03.919286+00:00 (in 29.999559 seconds)
buildlogsarchiver stdout | 2025-02-07 03:41:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:03 UTC)" (scheduled at 2025-02-07 03:41:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:41:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 41, 33, 919977), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:41:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:41:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:41:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:41:34,090 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:56.713547+00:00 (in 18.997204 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:37 UTC)" (scheduled at 2025-02-07 03:41:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:41:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:41:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:48.732147+00:00 (in 9.999412 seconds)
notificationworker stdout | 2025-02-07 03:41:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:48 UTC)" (scheduled at 2025-02-07 03:41:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:41:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:41:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 38, 733113), True, datetime.datetime(2025, 2, 7, 3, 41, 38, 733113), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:41:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:41:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:41:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:41:38,770 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:41:39,272 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:41:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:41:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:54.423055+00:00 (in 13.998324 seconds)
gcworker stdout | 2025-02-07 03:41:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:10 UTC)" (scheduled at 2025-02-07 03:41:40.424250+00:00)
gcworker stdout | 2025-02-07 03:41:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:41:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899400430, None, 1, 0])
gcworker stdout | 2025-02-07 03:41:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:41:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:41:40,448 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:41:41,345 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:41:41,435 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:41:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:41:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:11.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 03:41:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:11 UTC)" (scheduled at 2025-02-07 03:41:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:41:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896101836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:41:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:41:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:41:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:41:42,486 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:42,792 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:41:43,482 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:41:43,583 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:41:43,612 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:41:43,971 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:41:44,236 [238] [DEBUG] [app] Starting request: urn:request:e8ca8032-02b0-4732-bd08-ff5789609891 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:44,236 [236] [DEBUG] [app] Starting request: urn:request:6914336d-1b6c-4349-a86f-e1176c597bd1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:41:44,239 [246] [DEBUG] [app] Starting request: urn:request:f761a931-c0db-4a83-aab4-61b5a7c5fb81 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:44,239 [245] [DEBUG] [app] Starting request: urn:request:892b666d-de64-40a8-8757-875cdfb4b2d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:44,239 [246] [DEBUG] [app] Ending request: urn:request:f761a931-c0db-4a83-aab4-61b5a7c5fb81 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f761a931-c0db-4a83-aab4-61b5a7c5fb81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:41:44,239 [245] [DEBUG] [app] Ending request: urn:request:892b666d-de64-40a8-8757-875cdfb4b2d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:892b666d-de64-40a8-8757-875cdfb4b2d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:41:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:41:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:41:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:44,242 [238] [DEBUG] [app] Starting request: urn:request:4a18e745-b080-4d0c-8c79-76838ced0c06 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:44,242 [235] [DEBUG] [app] Starting request: urn:request:6e2a95b4-d6ea-4d24-90ed-53ec11659f1f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:44,242 [238] [DEBUG] [app] Ending request: urn:request:4a18e745-b080-4d0c-8c79-76838ced0c06 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a18e745-b080-4d0c-8c79-76838ced0c06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:41:44,242 [235] [DEBUG] [app] Ending request: urn:request:6e2a95b4-d6ea-4d24-90ed-53ec11659f1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e2a95b4-d6ea-4d24-90ed-53ec11659f1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:41:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:41:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:41:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:41:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:41:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:44,254 [236] [DEBUG] [app] Ending request: urn:request:6914336d-1b6c-4349-a86f-e1176c597bd1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6914336d-1b6c-4349-a86f-e1176c597bd1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:41:44,254 [238] [DEBUG] [app] Ending request: urn:request:e8ca8032-02b0-4732-bd08-ff5789609891 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e8ca8032-02b0-4732-bd08-ff5789609891', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 03:41:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:41:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gcworker stdout | 2025-02-07 03:41:44,511 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:41:44,593 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:41:44,639 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:41:45,967 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:41:45,970 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:41:45,972 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:41:45,978 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:41:45,982 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:41:46,274 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:41:46,610 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:41:46,790 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:41:46,997 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:41:47,000 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:41:47,002 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:41:47,967 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:41:47,971 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:41:47,977 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:41:47,980 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:41:47,985 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:41:47,990 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:41:47,995 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:41:47,998 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:41:48,000 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:41:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:41:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:50.814677+00:00 (in 2.082067 seconds)
notificationworker stdout | 2025-02-07 03:41:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:58 UTC)" (scheduled at 2025-02-07 03:41:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:41:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:41:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 48, 732886), True, datetime.datetime(2025, 2, 7, 3, 41, 48, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:41:48,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:41:48,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:41:48,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:41:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:41:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:41:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:41:58.732147+00:00 (in 7.917018 seconds)
notificationworker stdout | 2025-02-07 03:41:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:50 UTC)" (scheduled at 2025-02-07 03:41:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:41:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:41:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:50 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 03:41:52,134 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2025-02-07 03:41:52,135 [68] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-02-07 03:41:52 UTC)" (scheduled at 2025-02-07 03:41:52.134599+00:00)
manifestbackfillworker stdout | 2025-02-07 03:41:52,135 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0])
manifestbackfillworker stdout | 2025-02-07 03:41:52,136 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:52.134599+00:00 (in 3599.998535 seconds)
manifestbackfillworker stdout | 2025-02-07 03:41:52,142 [68] [DEBUG] [__main__] Manifest backfill worker has completed; skipping
manifestbackfillworker stdout | 2025-02-07 03:41:52,142 [68] [DEBUG] [data.database] Disconnecting from database.
manifestbackfillworker stdout | 2025-02-07 03:41:52,142 [68] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-02-07 04:41:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:41:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:41:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:10.424250+00:00 (in 16.000715 seconds)
gcworker stdout | 2025-02-07 03:41:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:24 UTC)" (scheduled at 2025-02-07 03:41:54.423055+00:00)
gcworker stdout | 2025-02-07 03:41:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:41:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:41:55,836 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:41:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:41:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:56.118855+00:00 (in 59.999516 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:41:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:56 UTC)" (scheduled at 2025-02-07 03:41:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:41:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:41:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:41:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:37.715878+00:00 (in 41.001886 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:56 UTC)" (scheduled at 2025-02-07 03:41:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:41:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:41:57,818 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:41:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:41:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:08.732147+00:00 (in 9.999587 seconds)
notificationworker stdout | 2025-02-07 03:41:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:08 UTC)" (scheduled at 2025-02-07 03:41:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:41:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:41:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 41, 58, 732821), True, datetime.datetime(2025, 2, 7, 3, 41, 58, 732821), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:41:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:41:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:41:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:41:58,832 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:41:59,235 [238] [DEBUG] [app] Starting request: urn:request:28de4dd8-6571-4c0b-a6c5-3bf20fdc1833 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:59,235 [235] [DEBUG] [app] Starting request: urn:request:58ede8ca-dce9-4b3f-b195-a88a79dc64da (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:41:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:41:59,238 [245] [DEBUG] [app] Starting request: urn:request:d393bf9e-aa4a-4c83-bad7-b515470a6426 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:59,238 [239] [DEBUG] [app] Starting request: urn:request:707b3f12-09f7-4aee-98a1-290738766702 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:41:59,238 [239] [DEBUG] [app] Ending request: urn:request:707b3f12-09f7-4aee-98a1-290738766702 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:707b3f12-09f7-4aee-98a1-290738766702', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:41:59,238 [245] [DEBUG] [app] Ending request: urn:request:d393bf9e-aa4a-4c83-bad7-b515470a6426 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d393bf9e-aa4a-4c83-bad7-b515470a6426', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:41:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:41:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:41:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:41:59,241 [235] [DEBUG] [app] Starting request: urn:request:7d32c0b4-661d-43a9-87c1-b19b86da3e39 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:59,242 [235] [DEBUG] [app] Ending request: urn:request:7d32c0b4-661d-43a9-87c1-b19b86da3e39 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d32c0b4-661d-43a9-87c1-b19b86da3e39', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:41:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:41:59,242 [237] [DEBUG] [app] Starting request: urn:request:df557cef-064c-44b4-a947-3319d25493c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:41:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:59,242 [237] [DEBUG] [app] Ending request: urn:request:df557cef-064c-44b4-a947-3319d25493c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df557cef-064c-44b4-a947-3319d25493c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:41:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:41:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:41:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:41:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:41:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:41:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:41:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:41:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:41:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:41:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:41:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:41:59,254 [235] [DEBUG] [app] Ending request: urn:request:58ede8ca-dce9-4b3f-b195-a88a79dc64da (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:58ede8ca-dce9-4b3f-b195-a88a79dc64da', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:59,254 [238] [DEBUG] [app] Ending request: urn:request:28de4dd8-6571-4c0b-a6c5-3bf20fdc1833 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28de4dd8-6571-4c0b-a6c5-3bf20fdc1833', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:41:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:41:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:41:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:41:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:41:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:41:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:41:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:41:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:04.936688+00:00 (in 4.998269 seconds)
exportactionlogsworker stdout | 2025-02-07 03:41:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:59 UTC)" (scheduled at 2025-02-07 03:41:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:41:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:41:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:42:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:42:03,147 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:42:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:42:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:23.420886+00:00 (in 20.001306 seconds)
repositorygcworker stdout | 2025-02-07 03:42:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:03 UTC)" (scheduled at 2025-02-07 03:42:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:42:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:42:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 3, 419859), True, datetime.datetime(2025, 2, 7, 3, 42, 3, 419859), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:42:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:42:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:42:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:42:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:42:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:33.919286+00:00 (in 29.999559 seconds)
buildlogsarchiver stdout | 2025-02-07 03:42:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:33 UTC)" (scheduled at 2025-02-07 03:42:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:42:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 42, 3, 919979), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:42:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:42:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:42:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:42:04,117 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:42:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:42:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:59.937899+00:00 (in 55.000735 seconds)
exportactionlogsworker stdout | 2025-02-07 03:42:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:04 UTC)" (scheduled at 2025-02-07 03:42:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:42:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:42:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 4, 937408), True, datetime.datetime(2025, 2, 7, 3, 42, 4, 937408), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:42:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:42:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:42:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:42:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:18.732147+00:00 (in 9.999428 seconds)
notificationworker stdout | 2025-02-07 03:42:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:18 UTC)" (scheduled at 2025-02-07 03:42:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:42:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:42:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 8, 733100), True, datetime.datetime(2025, 2, 7, 3, 42, 8, 733100), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:42:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:42:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:42:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:42:08,806 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:42:09,310 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:42:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:42:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:24.423055+00:00 (in 13.998334 seconds)
gcworker stdout | 2025-02-07 03:42:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:40 UTC)" (scheduled at 2025-02-07 03:42:10.424250+00:00)
gcworker stdout | 2025-02-07 03:42:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:42:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899430431, None, 1, 0])
gcworker stdout | 2025-02-07 03:42:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:42:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:42:10,457 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:42:11,354 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:42:11,471 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:42:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:42:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:41.831521+00:00 (in 29.999530 seconds)
autopruneworker stdout | 2025-02-07 03:42:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:41 UTC)" (scheduled at 2025-02-07 03:42:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:42:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896131836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:42:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:42:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:42:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:42:12,518 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:12,828 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:42:13,512 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:42:13,618 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
namespacegcworker stdout | 2025-02-07 03:42:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:42:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:31.621131+00:00 (in 17.996390 seconds)
namespacegcworker stdout | 2025-02-07 03:42:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:47:13 UTC)" (scheduled at 2025-02-07 03:42:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:42:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 42, 13, 624917), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:42:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 42, 13, 624917), True, datetime.datetime(2025, 2, 7, 3, 42, 13, 624917), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:42:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 42, 13, 624917), True, datetime.datetime(2025, 2, 7, 3, 42, 13, 624917), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 42, 13, 624917), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:42:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:42:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:47:13 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 03:42:13,647 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:42:14,008 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:42:14,235 [235] [DEBUG] [app] Starting request: urn:request:83d465a8-42ff-4328-8350-4012cfb869d9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:14,235 [237] [DEBUG] [app] Starting request: urn:request:1ac56833-f596-49e6-be64-34e933aad06d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:42:14,238 [244] [DEBUG] [app] Starting request: urn:request:2cb4fe8f-42e4-47d9-9d2d-8c6a57eb5000 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:14,238 [246] [DEBUG] [app] Starting request: urn:request:0819a094-5c15-4eff-9921-7826777bf7af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:14,239 [244] [DEBUG] [app] Ending request: urn:request:2cb4fe8f-42e4-47d9-9d2d-8c6a57eb5000 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2cb4fe8f-42e4-47d9-9d2d-8c6a57eb5000', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:42:14,239 [246] [DEBUG] [app] Ending request: urn:request:0819a094-5c15-4eff-9921-7826777bf7af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0819a094-5c15-4eff-9921-7826777bf7af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:42:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:42:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:14,242 [238] [DEBUG] [app] Starting request: urn:request:302cb96b-95dd-49c7-9fe5-5b5d2aa61ea1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:14,242 [236] [DEBUG] [app] Starting request: urn:request:fe03185d-2c96-4e82-86d9-f6cd5fce4358 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:14,242 [238] [DEBUG] [app] Ending request: urn:request:302cb96b-95dd-49c7-9fe5-5b5d2aa61ea1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:302cb96b-95dd-49c7-9fe5-5b5d2aa61ea1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:42:14,243 [236] [DEBUG] [app] Ending request: urn:request:fe03185d-2c96-4e82-86d9-f6cd5fce4358 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe03185d-2c96-4e82-86d9-f6cd5fce4358', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:42:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:42:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:42:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:42:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:42:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:14,254 [235] [DEBUG] [app] Ending request: urn:request:83d465a8-42ff-4328-8350-4012cfb869d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83d465a8-42ff-4328-8350-4012cfb869d9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:14,255 [237] [DEBUG] [app] Ending request: urn:request:1ac56833-f596-49e6-be64-34e933aad06d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ac56833-f596-49e6-be64-34e933aad06d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:42:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:42:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:42:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gcworker stdout | 2025-02-07 03:42:14,537 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:42:14,607 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:42:14,658 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:42:15,974 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:42:15,978 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:42:15,981 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:42:15,986 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:42:15,989 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:42:16,303 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:42:16,618 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:42:16,826 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:42:17,004 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:42:17,008 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:42:17,011 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:42:17,977 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:42:17,980 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:42:17,987 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:42:17,990 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:42:17,993 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:42:18,001 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:42:18,004 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:42:18,007 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:42:18,009 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:42:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:28.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 03:42:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:28 UTC)" (scheduled at 2025-02-07 03:42:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:42:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:42:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 18, 732885), True, datetime.datetime(2025, 2, 7, 3, 42, 18, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:42:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:42:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:42:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:42:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:42:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:03.419110+00:00 (in 39.997744 seconds)
repositorygcworker stdout | 2025-02-07 03:42:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:23 UTC)" (scheduled at 2025-02-07 03:42:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:42:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:42:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:42:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:42:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:40.424250+00:00 (in 16.000749 seconds)
gcworker stdout | 2025-02-07 03:42:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:54 UTC)" (scheduled at 2025-02-07 03:42:24.423055+00:00)
gcworker stdout | 2025-02-07 03:42:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:42:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:42:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:42:25,851 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:42:27,843 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:42:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:33.814166+00:00 (in 5.081566 seconds)
notificationworker stdout | 2025-02-07 03:42:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:38 UTC)" (scheduled at 2025-02-07 03:42:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:42:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:42:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 28, 732885), True, datetime.datetime(2025, 2, 7, 3, 42, 28, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:42:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:42:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:42:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:42:28,868 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:42:29,235 [238] [DEBUG] [app] Starting request: urn:request:657ebe2a-a126-4be3-a02f-027d327274f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:29,235 [236] [DEBUG] [app] Starting request: urn:request:2feaf897-16ff-4744-acbf-68a6d22e2ff8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:42:29,238 [246] [DEBUG] [app] Starting request: urn:request:664a89b5-0c36-45d2-bc02-fc4ddf551ce3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:29,239 [246] [DEBUG] [app] Ending request: urn:request:664a89b5-0c36-45d2-bc02-fc4ddf551ce3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:664a89b5-0c36-45d2-bc02-fc4ddf551ce3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:42:29,238 [239] [DEBUG] [app] Starting request: urn:request:6193e0f9-5ab4-4d19-bf2a-c2bbf1460eda (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:42:29,239 [239] [DEBUG] [app] Ending request: urn:request:6193e0f9-5ab4-4d19-bf2a-c2bbf1460eda (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6193e0f9-5ab4-4d19-bf2a-c2bbf1460eda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:42:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:42:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:29,242 [235] [DEBUG] [app] Starting request: urn:request:5895d6be-5840-4ab0-8d68-002a656fb367 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:29,242 [235] [DEBUG] [app] Ending request: urn:request:5895d6be-5840-4ab0-8d68-002a656fb367 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5895d6be-5840-4ab0-8d68-002a656fb367', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:42:29,242 [237] [DEBUG] [app] Starting request: urn:request:f92e81c4-72c5-4ec7-a2d9-dd44f05ee2e3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:29,243 [237] [DEBUG] [app] Ending request: urn:request:f92e81c4-72c5-4ec7-a2d9-dd44f05ee2e3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f92e81c4-72c5-4ec7-a2d9-dd44f05ee2e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:42:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:42:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:42:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:42:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:29,254 [236] [DEBUG] [app] Ending request: urn:request:2feaf897-16ff-4744-acbf-68a6d22e2ff8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2feaf897-16ff-4744-acbf-68a6d22e2ff8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:42:29,254 [238] [DEBUG] [app] Ending request: urn:request:657ebe2a-a126-4be3-a02f-027d327274f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:657ebe2a-a126-4be3-a02f-027d327274f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:42:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:42:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:42:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:42:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:32.625945+00:00 (in 1.004350 seconds)
namespacegcworker stdout | 2025-02-07 03:42:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:31 UTC)" (scheduled at 2025-02-07 03:42:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:42:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:42:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 31, 621877), True, datetime.datetime(2025, 2, 7, 3, 42, 31, 621877), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:42:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:42:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:42:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:42:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:42:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:31.621131+00:00 (in 58.994720 seconds)
namespacegcworker stdout | 2025-02-07 03:42:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:32 UTC)" (scheduled at 2025-02-07 03:42:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:42:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:42:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:42:33,177 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 03:42:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:38.732147+00:00 (in 4.917523 seconds)
notificationworker stdout | 2025-02-07 03:42:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:47:33 UTC)" (scheduled at 2025-02-07 03:42:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:42:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 42, 33, 814889), 'notification/%'])
notificationworker stdout | 2025-02-07 03:42:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 42, 33, 814889), True, datetime.datetime(2025, 2, 7, 3, 42, 33, 814889), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:42:33,824 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 42, 33, 814889), True, datetime.datetime(2025, 2, 7, 3, 42, 33, 814889), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 42, 33, 814889), 'notification/%'])
notificationworker stdout | 2025-02-07 03:42:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:42:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:47:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:42:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:42:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:03.919286+00:00 (in 29.999540 seconds)
buildlogsarchiver stdout | 2025-02-07 03:42:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:03 UTC)" (scheduled at 2025-02-07 03:42:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:42:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 42, 33, 920006), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:42:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:42:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:42:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:42:34,141 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:56.713547+00:00 (in 18.997177 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:37 UTC)" (scheduled at 2025-02-07 03:42:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:42:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:48.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 03:42:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:48 UTC)" (scheduled at 2025-02-07 03:42:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:42:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:42:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 38, 732845), True, datetime.datetime(2025, 2, 7, 3, 42, 38, 732845), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:42:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:42:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:42:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:42:38,831 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:42:39,343 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:42:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:42:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:54.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 03:42:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:10 UTC)" (scheduled at 2025-02-07 03:42:40.424250+00:00)
gcworker stdout | 2025-02-07 03:42:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:42:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899460431, None, 1, 0])
gcworker stdout | 2025-02-07 03:42:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:42:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:42:40,493 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:42:41,390 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:42:41,498 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:42:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:42:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:11.831521+00:00 (in 29.999558 seconds)
autopruneworker stdout | 2025-02-07 03:42:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:11 UTC)" (scheduled at 2025-02-07 03:42:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:42:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896161836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:42:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:42:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:42:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:42:42,538 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:42,858 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:42:43,530 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:42:43,654 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:42:43,683 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:42:44,026 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:42:44,236 [235] [DEBUG] [app] Starting request: urn:request:08842b47-f304-4d52-90ee-47a84ad6ec9f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:44,236 [238] [DEBUG] [app] Starting request: urn:request:a0c6b43e-4eca-443d-9ce2-6ae87cceccc8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:44,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:42:44,240 [239] [DEBUG] [app] Starting request: urn:request:a5e0e732-f219-4983-9c4f-f02095cbd386 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:44,241 [243] [DEBUG] [app] Starting request: urn:request:281b5674-e018-426a-b668-280eb05d1e24 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:44,241 [239] [DEBUG] [app] Ending request: urn:request:a5e0e732-f219-4983-9c4f-f02095cbd386 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5e0e732-f219-4983-9c4f-f02095cbd386', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:42:44,241 [243] [DEBUG] [app] Ending request: urn:request:281b5674-e018-426a-b668-280eb05d1e24 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:281b5674-e018-426a-b668-280eb05d1e24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:42:44,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:42:44,241 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:44,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:42:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:44,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:44,244 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:44,244 [235] [DEBUG] [app] Starting request: urn:request:7278a091-ad51-4f3b-b70a-1a547ab4b058 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:44,245 [235] [DEBUG] [app] Ending request: urn:request:7278a091-ad51-4f3b-b70a-1a547ab4b058 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7278a091-ad51-4f3b-b70a-1a547ab4b058', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:42:44,245 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:42:44,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:44,246 [237] [DEBUG] [app] Starting request: urn:request:1f3120ee-a3f1-4dd5-9b99-d90592feaa18 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:44,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:44,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:42:44,246 [237] [DEBUG] [app] Ending request: urn:request:1f3120ee-a3f1-4dd5-9b99-d90592feaa18 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1f3120ee-a3f1-4dd5-9b99-d90592feaa18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:42:44,247 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:44,247 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:44,247 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:44,248 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:42:44,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:44,252 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:44,255 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:44,256 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:44,257 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:44,257 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:44,258 [235] [DEBUG] [app] Ending request: urn:request:08842b47-f304-4d52-90ee-47a84ad6ec9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08842b47-f304-4d52-90ee-47a84ad6ec9f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:44,258 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 03:42:44,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:42:44,259 [238] [DEBUG] [app] Ending request: urn:request:a0c6b43e-4eca-443d-9ce2-6ae87cceccc8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0c6b43e-4eca-443d-9ce2-6ae87cceccc8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:44,259 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:42:44,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gcworker stdout | 2025-02-07 03:42:44,545 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:42:44,641 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:42:44,674 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:42:45,983 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:42:45,987 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:42:45,990 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:42:45,994 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:42:45,998 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:42:46,333 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:42:46,641 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:42:46,862 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:42:47,012 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:42:47,015 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:42:47,018 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:42:47,988 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:42:47,990 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:42:47,997 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:42:48,000 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:42:48,003 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:42:48,009 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:42:48,014 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:42:48,017 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:42:48,020 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:42:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:50.814677+00:00 (in 2.082079 seconds)
notificationworker stdout | 2025-02-07 03:42:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:58 UTC)" (scheduled at 2025-02-07 03:42:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:42:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:42:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 48, 732877), True, datetime.datetime(2025, 2, 7, 3, 42, 48, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:42:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:42:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:42:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:42:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:42:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:42:58.732147+00:00 (in 7.917035 seconds)
notificationworker stdout | 2025-02-07 03:42:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:50 UTC)" (scheduled at 2025-02-07 03:42:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:42:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:42:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:42:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:42:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:10.424250+00:00 (in 16.000729 seconds)
gcworker stdout | 2025-02-07 03:42:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:24 UTC)" (scheduled at 2025-02-07 03:42:54.423055+00:00)
gcworker stdout | 2025-02-07 03:42:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:42:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:42:55,863 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:42:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:42:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:56.118855+00:00 (in 59.999520 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:42:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:56 UTC)" (scheduled at 2025-02-07 03:42:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:42:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:42:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:42:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:37.715878+00:00 (in 41.001872 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:56 UTC)" (scheduled at 2025-02-07 03:42:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:42:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:42:57,854 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:42:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:42:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:08.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 03:42:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:08 UTC)" (scheduled at 2025-02-07 03:42:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:42:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:42:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 42, 58, 732859), True, datetime.datetime(2025, 2, 7, 3, 42, 58, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:42:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:42:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:42:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:42:58,878 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:42:59,236 [235] [DEBUG] [app] Starting request: urn:request:cffbd248-240e-4b01-b9c5-9b185b602db7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:59,236 [237] [DEBUG] [app] Starting request: urn:request:a9394f67-1363-4039-9eef-6a2372bf6464 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:42:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:42:59,239 [246] [DEBUG] [app] Starting request: urn:request:dba40b39-658e-41f8-a6f2-68ee404e8d9e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:59,239 [245] [DEBUG] [app] Starting request: urn:request:a4066f72-6491-4067-9288-fad70f112c15 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:42:59,239 [246] [DEBUG] [app] Ending request: urn:request:dba40b39-658e-41f8-a6f2-68ee404e8d9e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dba40b39-658e-41f8-a6f2-68ee404e8d9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:42:59,239 [245] [DEBUG] [app] Ending request: urn:request:a4066f72-6491-4067-9288-fad70f112c15 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4066f72-6491-4067-9288-fad70f112c15', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:42:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:42:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:42:59,242 [235] [DEBUG] [app] Starting request: urn:request:4912c9e2-cda8-4a06-8de8-d24fd9594c4d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:59,242 [235] [DEBUG] [app] Ending request: urn:request:4912c9e2-cda8-4a06-8de8-d24fd9594c4d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4912c9e2-cda8-4a06-8de8-d24fd9594c4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:42:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:42:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:59,243 [238] [DEBUG] [app] Starting request: urn:request:c445f8cd-b44e-4868-abbd-33a6ed994910 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:42:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:42:59,243 [238] [DEBUG] [app] Ending request: urn:request:c445f8cd-b44e-4868-abbd-33a6ed994910 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c445f8cd-b44e-4868-abbd-33a6ed994910', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:42:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:42:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:42:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:42:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:42:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:42:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:42:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:42:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:42:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:42:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:42:59,253 [235] [DEBUG] [app] Ending request: urn:request:cffbd248-240e-4b01-b9c5-9b185b602db7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cffbd248-240e-4b01-b9c5-9b185b602db7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:42:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:42:59,254 [237] [DEBUG] [app] Ending request: urn:request:a9394f67-1363-4039-9eef-6a2372bf6464 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a9394f67-1363-4039-9eef-6a2372bf6464', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:42:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:42:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:42:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:42:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 03:42:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:42:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:04.936688+00:00 (in 4.998289 seconds)
exportactionlogsworker stdout | 2025-02-07 03:42:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:59 UTC)" (scheduled at 2025-02-07 03:42:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:42:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:42:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:43:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:43:03,213 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:43:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:43:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:23.420886+00:00 (in 20.001309 seconds)
repositorygcworker stdout | 2025-02-07 03:43:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:03 UTC)" (scheduled at 2025-02-07 03:43:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:43:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:43:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 3, 419852), True, datetime.datetime(2025, 2, 7, 3, 43, 3, 419852), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:43:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:43:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:43:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:43:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:43:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:33.919286+00:00 (in 29.999530 seconds)
buildlogsarchiver stdout | 2025-02-07 03:43:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:33 UTC)" (scheduled at 2025-02-07 03:43:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:43:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 43, 3, 920020), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:43:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:43:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:43:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:43:04,177 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:43:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:43:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:59.937899+00:00 (in 55.000746 seconds)
exportactionlogsworker stdout | 2025-02-07 03:43:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:04 UTC)" (scheduled at 2025-02-07 03:43:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:43:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:43:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 4, 937407), True, datetime.datetime(2025, 2, 7, 3, 43, 4, 937407), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:43:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:43:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:43:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:43:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:43:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:18.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 03:43:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:18 UTC)" (scheduled at 2025-02-07 03:43:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:43:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:43:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 8, 732862), True, datetime.datetime(2025, 2, 7, 3, 43, 8, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:43:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:43:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:43:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:43:08,854 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:43:09,379 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:43:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:43:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:24.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 03:43:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:40 UTC)" (scheduled at 2025-02-07 03:43:10.424250+00:00)
gcworker stdout | 2025-02-07 03:43:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:43:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899490430, None, 1, 0])
gcworker stdout | 2025-02-07 03:43:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:43:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:43:10,529 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:43:11,407 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:43:11,532 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:43:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:43:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:41.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 03:43:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:41 UTC)" (scheduled at 2025-02-07 03:43:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:43:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896191836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:43:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:43:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:43:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:43:12,574 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:12,894 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:43:13,566 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:43:13,691 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:43:13,699 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:43:14,041 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:43:14,235 [237] [DEBUG] [app] Starting request: urn:request:aa4a1c07-db81-44f7-87fe-d673b04dcd0f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:14,235 [236] [DEBUG] [app] Starting request: urn:request:8778a2f2-a936-4258-982d-6d5c998e5be3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:43:14,238 [245] [DEBUG] [app] Starting request: urn:request:560c7c45-97ab-4883-a2f6-4b63bda8117e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:14,238 [239] [DEBUG] [app] Starting request: urn:request:0a0dd8da-990d-445d-bf56-d9052f6a169f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:14,239 [239] [DEBUG] [app] Ending request: urn:request:0a0dd8da-990d-445d-bf56-d9052f6a169f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a0dd8da-990d-445d-bf56-d9052f6a169f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:43:14,239 [245] [DEBUG] [app] Ending request: urn:request:560c7c45-97ab-4883-a2f6-4b63bda8117e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:560c7c45-97ab-4883-a2f6-4b63bda8117e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:43:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:43:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:43:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:14,242 [237] [DEBUG] [app] Starting request: urn:request:34ce2876-00fd-4110-b405-d6d9898414ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:14,242 [237] [DEBUG] [app] Ending request: urn:request:34ce2876-00fd-4110-b405-d6d9898414ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:34ce2876-00fd-4110-b405-d6d9898414ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:43:14,242 [235] [DEBUG] [app] Starting request: urn:request:85d26697-fa60-4701-b644-cc5234ba136c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:14,243 [235] [DEBUG] [app] Ending request: urn:request:85d26697-fa60-4701-b644-cc5234ba136c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:85d26697-fa60-4701-b644-cc5234ba136c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:43:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:43:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:43:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:14,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:14,255 [237] [DEBUG] [app] Ending request: urn:request:aa4a1c07-db81-44f7-87fe-d673b04dcd0f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aa4a1c07-db81-44f7-87fe-d673b04dcd0f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:14,255 [236] [DEBUG] [app] Ending request: urn:request:8778a2f2-a936-4258-982d-6d5c998e5be3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8778a2f2-a936-4258-982d-6d5c998e5be3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:43:14,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gcworker stdout | 2025-02-07 03:43:14,581 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:43:14,677 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:43:14,710 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:43:15,991 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:43:15,996 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:43:15,999 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:43:16,003 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:43:16,006 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:43:16,357 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:43:16,677 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:43:16,898 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:43:17,021 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:43:17,024 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:43:17,027 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:43:17,999 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:43:18,003 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:43:18,006 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:43:18,009 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:43:18,011 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:43:18,017 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:43:18,024 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:43:18,027 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:43:18,029 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:43:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:43:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:28.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 03:43:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:28 UTC)" (scheduled at 2025-02-07 03:43:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:43:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:43:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 18, 732890), True, datetime.datetime(2025, 2, 7, 3, 43, 18, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:43:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:43:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:43:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:43:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:43:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:32.420707+00:00 (in 8.999346 seconds)
repositorygcworker stdout | 2025-02-07 03:43:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:23 UTC)" (scheduled at 2025-02-07 03:43:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:43:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:43:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:43:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:43:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:40.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 03:43:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:54 UTC)" (scheduled at 2025-02-07 03:43:24.423055+00:00)
gcworker stdout | 2025-02-07 03:43:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:43:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:43:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:43:25,895 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:43:27,872 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:43:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:43:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:38.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:43:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:38 UTC)" (scheduled at 2025-02-07 03:43:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:43:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:43:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 28, 732862), True, datetime.datetime(2025, 2, 7, 3, 43, 28, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:43:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:43:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:43:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:43:28,910 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:43:29,236 [235] [DEBUG] [app] Starting request: urn:request:3460eb3a-b6a7-4491-ac79-8a181c510555 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:29,236 [237] [DEBUG] [app] Starting request: urn:request:283eb195-c501-4c84-ab75-aef45bd3c528 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:43:29,238 [239] [DEBUG] [app] Starting request: urn:request:9d777012-6236-4796-ae48-ad99ca838ea9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:29,239 [245] [DEBUG] [app] Starting request: urn:request:51233768-9b08-414a-8df5-98765fa174c5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:29,239 [239] [DEBUG] [app] Ending request: urn:request:9d777012-6236-4796-ae48-ad99ca838ea9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d777012-6236-4796-ae48-ad99ca838ea9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:43:29,239 [245] [DEBUG] [app] Ending request: urn:request:51233768-9b08-414a-8df5-98765fa174c5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51233768-9b08-414a-8df5-98765fa174c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:43:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:43:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:43:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:29,242 [237] [DEBUG] [app] Starting request: urn:request:c0b2d4cd-a509-45a2-930c-7c75cf633d20 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:29,242 [236] [DEBUG] [app] Starting request: urn:request:98cfcbd7-6ae8-4e2b-be67-921aa9bf8acc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:29,242 [237] [DEBUG] [app] Ending request: urn:request:c0b2d4cd-a509-45a2-930c-7c75cf633d20 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c0b2d4cd-a509-45a2-930c-7c75cf633d20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:29,242 [236] [DEBUG] [app] Ending request: urn:request:98cfcbd7-6ae8-4e2b-be67-921aa9bf8acc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98cfcbd7-6ae8-4e2b-be67-921aa9bf8acc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:43:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:43:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:43:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:43:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:29,254 [237] [DEBUG] [app] Ending request: urn:request:283eb195-c501-4c84-ab75-aef45bd3c528 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:283eb195-c501-4c84-ab75-aef45bd3c528', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:29,254 [235] [DEBUG] [app] Ending request: urn:request:3460eb3a-b6a7-4491-ac79-8a181c510555 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3460eb3a-b6a7-4491-ac79-8a181c510555', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:43:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 03:43:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:43:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:32.625945+00:00 (in 1.004408 seconds)
namespacegcworker stdout | 2025-02-07 03:43:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:31 UTC)" (scheduled at 2025-02-07 03:43:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:43:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:43:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 31, 621747), True, datetime.datetime(2025, 2, 7, 3, 43, 31, 621747), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:43:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:43:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:43:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:43:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:43:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:03.419110+00:00 (in 30.997943 seconds)
repositorygcworker stdout | 2025-02-07 03:43:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:48:32 UTC)" (scheduled at 2025-02-07 03:43:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:43:32,422 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 43, 32, 421469), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:43:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 43, 32, 421469), True, datetime.datetime(2025, 2, 7, 3, 43, 32, 421469), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:43:32,431 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 43, 32, 421469), True, datetime.datetime(2025, 2, 7, 3, 43, 32, 421469), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 43, 32, 421469), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:43:32,432 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:43:32,432 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:48:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:43:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:43:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:31.621131+00:00 (in 58.994688 seconds)
namespacegcworker stdout | 2025-02-07 03:43:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:32 UTC)" (scheduled at 2025-02-07 03:43:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:43:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:43:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:43:33,249 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:43:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:43:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:03.919286+00:00 (in 29.999513 seconds)
buildlogsarchiver stdout | 2025-02-07 03:43:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:03 UTC)" (scheduled at 2025-02-07 03:43:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:43:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 43, 33, 920072), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:43:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:43:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:43:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:43:34,207 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:56.713547+00:00 (in 18.997155 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:37 UTC)" (scheduled at 2025-02-07 03:43:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:43:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:43:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:48.732147+00:00 (in 9.999534 seconds)
notificationworker stdout | 2025-02-07 03:43:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:48 UTC)" (scheduled at 2025-02-07 03:43:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:43:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:43:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 38, 732906), True, datetime.datetime(2025, 2, 7, 3, 43, 38, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:43:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:43:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:43:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:43:38,888 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:43:39,387 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:43:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:43:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:54.423055+00:00 (in 13.998376 seconds)
gcworker stdout | 2025-02-07 03:43:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:10 UTC)" (scheduled at 2025-02-07 03:43:40.424250+00:00)
gcworker stdout | 2025-02-07 03:43:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:43:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899520431, None, 1, 0])
gcworker stdout | 2025-02-07 03:43:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:43:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:43:40,565 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:43:41,442 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:43:41,568 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:43:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:43:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:11.831521+00:00 (in 29.999519 seconds)
autopruneworker stdout | 2025-02-07 03:43:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:11 UTC)" (scheduled at 2025-02-07 03:43:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:43:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896221836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:43:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:43:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:43:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:43:42,606 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:42,930 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:43:43,579 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:43:43,706 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:43:43,734 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:43:44,064 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:43:44,236 [235] [DEBUG] [app] Starting request: urn:request:256bc0f8-52b8-4d2a-b500-1edbf06950f5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:44,236 [237] [DEBUG] [app] Starting request: urn:request:595eb038-04ee-4bd7-9140-c2a7729bdbb8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:43:44,239 [239] [DEBUG] [app] Starting request: urn:request:8fcfa82c-62cc-4eee-bc16-d97212335130 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:44,239 [245] [DEBUG] [app] Starting request: urn:request:dde43743-9bc0-4354-8205-2801be5e8f45 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:44,239 [239] [DEBUG] [app] Ending request: urn:request:8fcfa82c-62cc-4eee-bc16-d97212335130 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fcfa82c-62cc-4eee-bc16-d97212335130', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:43:44,239 [245] [DEBUG] [app] Ending request: urn:request:dde43743-9bc0-4354-8205-2801be5e8f45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dde43743-9bc0-4354-8205-2801be5e8f45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:43:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:43:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:43:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 03:43:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:44,242 [236] [DEBUG] [app] Starting request: urn:request:fd62c433-e78c-4fdf-a9c0-70041627dbae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:44,242 [235] [DEBUG] [app] Starting request: urn:request:64b8d4e0-b04f-4314-8461-68c5a1fd296e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:44,243 [236] [DEBUG] [app] Ending request: urn:request:fd62c433-e78c-4fdf-a9c0-70041627dbae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd62c433-e78c-4fdf-a9c0-70041627dbae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:44,243 [235] [DEBUG] [app] Ending request: urn:request:64b8d4e0-b04f-4314-8461-68c5a1fd296e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64b8d4e0-b04f-4314-8461-68c5a1fd296e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:43:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:43:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:43:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:43:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:44,255 [237] [DEBUG] [app] Ending request: urn:request:595eb038-04ee-4bd7-9140-c2a7729bdbb8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:595eb038-04ee-4bd7-9140-c2a7729bdbb8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:44,255 [235] [DEBUG] [app] Ending request: urn:request:256bc0f8-52b8-4d2a-b500-1edbf06950f5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:256bc0f8-52b8-4d2a-b500-1edbf06950f5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:43:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gcworker stdout | 2025-02-07 03:43:44,618 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:43:44,715 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:43:44,747 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:43:45,999 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:43:46,004 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:43:46,006 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:43:46,009 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:43:46,013 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:43:46,394 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:43:46,710 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:43:46,914 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:43:47,029 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:43:47,031 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:43:47,035 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:43:48,010 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:43:48,013 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:43:48,016 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:43:48,018 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:43:48,021 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:43:48,025 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:43:48,031 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:43:48,033 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:43:48,039 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:43:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:43:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:50.814677+00:00 (in 2.082091 seconds)
notificationworker stdout | 2025-02-07 03:43:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:58 UTC)" (scheduled at 2025-02-07 03:43:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:43:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:43:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 48, 732872), True, datetime.datetime(2025, 2, 7, 3, 43, 48, 732872), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:43:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:43:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:43:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:43:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:43:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:43:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:43:58.732147+00:00 (in 7.916996 seconds)
notificationworker stdout | 2025-02-07 03:43:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:50 UTC)" (scheduled at 2025-02-07 03:43:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:43:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:43:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:43:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:43:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:10.424250+00:00 (in 16.000684 seconds)
gcworker stdout | 2025-02-07 03:43:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:24 UTC)" (scheduled at 2025-02-07 03:43:54.423055+00:00)
gcworker stdout | 2025-02-07 03:43:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:43:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:43:55,914 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:43:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:43:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:56.118855+00:00 (in 59.999472 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:43:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:56 UTC)" (scheduled at 2025-02-07 03:43:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:43:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:43:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:43:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:37.715878+00:00 (in 41.001841 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:56 UTC)" (scheduled at 2025-02-07 03:43:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:43:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:43:57,897 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:43:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:43:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:08.732147+00:00 (in 9.999480 seconds)
notificationworker stdout | 2025-02-07 03:43:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:08 UTC)" (scheduled at 2025-02-07 03:43:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:43:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:43:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 43, 58, 732950), True, datetime.datetime(2025, 2, 7, 3, 43, 58, 732950), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:43:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:43:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:43:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:43:58,932 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:43:59,235 [235] [DEBUG] [app] Starting request: urn:request:9b96df0c-7497-4809-ab47-108c0089cd93 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:59,235 [237] [DEBUG] [app] Starting request: urn:request:893b8d1e-e5c2-4de1-b4a7-2142b17191ab (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:43:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:43:59,238 [246] [DEBUG] [app] Starting request: urn:request:adfe7d4e-8ad4-4c63-85b9-a6ac8dbe3722 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:59,238 [239] [DEBUG] [app] Starting request: urn:request:3e2baaf3-a1f7-4033-b981-46a2fa6b3b1f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:43:59,239 [246] [DEBUG] [app] Ending request: urn:request:adfe7d4e-8ad4-4c63-85b9-a6ac8dbe3722 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:adfe7d4e-8ad4-4c63-85b9-a6ac8dbe3722', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:43:59,239 [239] [DEBUG] [app] Ending request: urn:request:3e2baaf3-a1f7-4033-b981-46a2fa6b3b1f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e2baaf3-a1f7-4033-b981-46a2fa6b3b1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:43:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:43:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:43:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:43:59,242 [235] [DEBUG] [app] Starting request: urn:request:3fe7e75f-afd0-4664-b2eb-a0adf589255a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:59,242 [235] [DEBUG] [app] Ending request: urn:request:3fe7e75f-afd0-4664-b2eb-a0adf589255a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3fe7e75f-afd0-4664-b2eb-a0adf589255a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:59,242 [237] [DEBUG] [app] Starting request: urn:request:3aa7804c-d8cf-4ce1-995d-508829db3294 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:43:59,243 [237] [DEBUG] [app] Ending request: urn:request:3aa7804c-d8cf-4ce1-995d-508829db3294 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3aa7804c-d8cf-4ce1-995d-508829db3294', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:43:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:43:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:43:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:43:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:43:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:43:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:43:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:43:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:43:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:43:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:43:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:43:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:43:59,255 [235] [DEBUG] [app] Ending request: urn:request:9b96df0c-7497-4809-ab47-108c0089cd93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b96df0c-7497-4809-ab47-108c0089cd93', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:59,255 [237] [DEBUG] [app] Ending request: urn:request:893b8d1e-e5c2-4de1-b4a7-2142b17191ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:893b8d1e-e5c2-4de1-b4a7-2142b17191ab', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:43:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:43:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:43:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:43:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:43:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 03:43:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:43:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:04.936688+00:00 (in 4.998221 seconds)
exportactionlogsworker stdout | 2025-02-07 03:43:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:59 UTC)" (scheduled at 2025-02-07 03:43:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:43:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:43:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:44:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:44:03,258 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:44:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:44:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:23.420886+00:00 (in 20.001315 seconds)
repositorygcworker stdout | 2025-02-07 03:44:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:03 UTC)" (scheduled at 2025-02-07 03:44:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:44:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:44:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 3, 419854), True, datetime.datetime(2025, 2, 7, 3, 44, 3, 419854), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:44:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:44:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:44:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:44:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:44:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:33.919286+00:00 (in 29.999489 seconds)
buildlogsarchiver stdout | 2025-02-07 03:44:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:33 UTC)" (scheduled at 2025-02-07 03:44:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:44:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 44, 3, 920058), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:44:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:44:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:44:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:44:04,243 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:44:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:44:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:52.937648+00:00 (in 48.000475 seconds)
exportactionlogsworker stdout | 2025-02-07 03:44:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:04 UTC)" (scheduled at 2025-02-07 03:44:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:44:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:44:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 4, 937428), True, datetime.datetime(2025, 2, 7, 3, 44, 4, 937428), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:44:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:44:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:44:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:44:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:44:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:18.732147+00:00 (in 9.999531 seconds)
notificationworker stdout | 2025-02-07 03:44:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:18 UTC)" (scheduled at 2025-02-07 03:44:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:44:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:44:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 8, 732930), True, datetime.datetime(2025, 2, 7, 3, 44, 8, 732930), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:44:08,744 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:44:08,744 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:44:08,744 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:44:08,924 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:44:09,418 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:44:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:44:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:24.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 03:44:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:40 UTC)" (scheduled at 2025-02-07 03:44:10.424250+00:00)
gcworker stdout | 2025-02-07 03:44:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:44:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899550431, None, 1, 0])
gcworker stdout | 2025-02-07 03:44:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:44:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:44:10,601 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:44:11,478 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:44:11,604 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:44:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:44:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:41.831521+00:00 (in 29.999543 seconds)
autopruneworker stdout | 2025-02-07 03:44:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:41 UTC)" (scheduled at 2025-02-07 03:44:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:44:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896251836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:44:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:44:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:44:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:44:12,642 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:12,967 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:44:13,616 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:44:13,743 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:44:13,771 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:44:14,099 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:44:14,236 [238] [DEBUG] [app] Starting request: urn:request:d1c315dc-946d-43f8-a5d4-9a135832a7e5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:14,236 [235] [DEBUG] [app] Starting request: urn:request:1e37d5c4-3f34-4b96-b19f-0ecd864dbb97 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:44:14,239 [245] [DEBUG] [app] Starting request: urn:request:bfb88d21-0f76-408c-bfd2-0a9cb389ae34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:14,239 [246] [DEBUG] [app] Starting request: urn:request:25687eca-2f09-4351-927d-4b591ad57723 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:14,239 [245] [DEBUG] [app] Ending request: urn:request:bfb88d21-0f76-408c-bfd2-0a9cb389ae34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bfb88d21-0f76-408c-bfd2-0a9cb389ae34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:44:14,239 [246] [DEBUG] [app] Ending request: urn:request:25687eca-2f09-4351-927d-4b591ad57723 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:25687eca-2f09-4351-927d-4b591ad57723', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:44:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:44:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:44:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:14,242 [235] [DEBUG] [app] Starting request: urn:request:1c78574d-6d58-496f-ba15-b25ccc173acd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:14,242 [238] [DEBUG] [app] Starting request: urn:request:040b7be8-6112-4fb9-92c4-0ccccdc2921e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:14,242 [235] [DEBUG] [app] Ending request: urn:request:1c78574d-6d58-496f-ba15-b25ccc173acd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1c78574d-6d58-496f-ba15-b25ccc173acd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:14,242 [238] [DEBUG] [app] Ending request: urn:request:040b7be8-6112-4fb9-92c4-0ccccdc2921e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:040b7be8-6112-4fb9-92c4-0ccccdc2921e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:44:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:14,255 [235] [DEBUG] [app] Ending request: urn:request:1e37d5c4-3f34-4b96-b19f-0ecd864dbb97 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e37d5c4-3f34-4b96-b19f-0ecd864dbb97', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:14,255 [238] [DEBUG] [app] Ending request: urn:request:d1c315dc-946d-43f8-a5d4-9a135832a7e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1c315dc-946d-43f8-a5d4-9a135832a7e5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:44:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:44:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:44:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gcworker stdout | 2025-02-07 03:44:14,641 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:44:14,731 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:44:14,780 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:44:16,006 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:44:16,011 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:44:16,013 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:44:16,016 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:44:16,020 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:44:16,430 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:44:16,717 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:44:16,939 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:44:17,037 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:44:17,041 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:44:17,044 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:44:18,019 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:44:18,023 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:44:18,027 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:44:18,030 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:44:18,032 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:44:18,035 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:44:18,038 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:44:18,041 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:44:18,048 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:44:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:44:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:28.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 03:44:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:28 UTC)" (scheduled at 2025-02-07 03:44:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:44:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:44:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 18, 732895), True, datetime.datetime(2025, 2, 7, 3, 44, 18, 732895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:44:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:44:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:44:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:44:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:44:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:03.419110+00:00 (in 39.997748 seconds)
repositorygcworker stdout | 2025-02-07 03:44:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:23 UTC)" (scheduled at 2025-02-07 03:44:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:44:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:44:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:44:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:44:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:40.424250+00:00 (in 16.000721 seconds)
gcworker stdout | 2025-02-07 03:44:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:54 UTC)" (scheduled at 2025-02-07 03:44:24.423055+00:00)
gcworker stdout | 2025-02-07 03:44:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:44:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:44:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:44:25,946 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:44:27,931 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:44:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:44:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:38.732147+00:00 (in 9.999436 seconds)
notificationworker stdout | 2025-02-07 03:44:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:38 UTC)" (scheduled at 2025-02-07 03:44:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:44:28,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:44:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 28, 733096), True, datetime.datetime(2025, 2, 7, 3, 44, 28, 733096), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:44:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:44:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:44:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:44:28,971 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:44:29,236 [235] [DEBUG] [app] Starting request: urn:request:e0e2cda2-6e38-4228-8652-6f84b5fd0f8b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:29,236 [236] [DEBUG] [app] Starting request: urn:request:bc52a672-f957-459f-b7aa-ec754b865fca (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:29,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:44:29,241 [246] [DEBUG] [app] Starting request: urn:request:319563f1-e1e9-44c7-aa7b-c2e1ac37eff7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:29,241 [243] [DEBUG] [app] Starting request: urn:request:04102d1a-f0de-4aac-a76d-e8d486a66783 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:29,241 [246] [DEBUG] [app] Ending request: urn:request:319563f1-e1e9-44c7-aa7b-c2e1ac37eff7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:319563f1-e1e9-44c7-aa7b-c2e1ac37eff7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:44:29,241 [243] [DEBUG] [app] Ending request: urn:request:04102d1a-f0de-4aac-a76d-e8d486a66783 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04102d1a-f0de-4aac-a76d-e8d486a66783', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:44:29,242 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 162 0.003)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:44:29,242 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:29,244 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:29,244 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:29,245 [235] [DEBUG] [app] Starting request: urn:request:1e4b265a-dfdc-46dd-98ed-da1eb9190e82 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:29,245 [237] [DEBUG] [app] Starting request: urn:request:005baf7d-0cae-445b-ba1d-cdda37d993fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:29,245 [235] [DEBUG] [app] Ending request: urn:request:1e4b265a-dfdc-46dd-98ed-da1eb9190e82 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e4b265a-dfdc-46dd-98ed-da1eb9190e82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:29,246 [237] [DEBUG] [app] Ending request: urn:request:005baf7d-0cae-445b-ba1d-cdda37d993fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:005baf7d-0cae-445b-ba1d-cdda37d993fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:29,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:44:29,246 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:29,246 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:29,246 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:29,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:29,247 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:29,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:29,247 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:29,252 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:29,252 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:29,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:29,257 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:29,258 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:29,259 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:29,259 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:29,260 [236] [DEBUG] [app] Ending request: urn:request:bc52a672-f957-459f-b7aa-ec754b865fca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bc52a672-f957-459f-b7aa-ec754b865fca', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:29,260 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:44:29,261 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.026 118 0.026)
gunicorn-web stdout | 2025-02-07 03:44:29,261 [235] [DEBUG] [app] Ending request: urn:request:e0e2cda2-6e38-4228-8652-6f84b5fd0f8b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0e2cda2-6e38-4228-8652-6f84b5fd0f8b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:29,261 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:44:29,262 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.027 118 0.027)
namespacegcworker stdout | 2025-02-07 03:44:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:44:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:32.625945+00:00 (in 1.004342 seconds)
namespacegcworker stdout | 2025-02-07 03:44:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:31 UTC)" (scheduled at 2025-02-07 03:44:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:44:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:44:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 31, 621880), True, datetime.datetime(2025, 2, 7, 3, 44, 31, 621880), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:44:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:44:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:44:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:44:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:44:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:31.621131+00:00 (in 58.994728 seconds)
namespacegcworker stdout | 2025-02-07 03:44:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:32 UTC)" (scheduled at 2025-02-07 03:44:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:44:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:44:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:44:33,270 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:44:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:44:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:03.919286+00:00 (in 29.999496 seconds)
buildlogsarchiver stdout | 2025-02-07 03:44:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:03 UTC)" (scheduled at 2025-02-07 03:44:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:44:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 44, 33, 920079), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:44:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:44:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:44:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:44:34,274 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:56.713547+00:00 (in 18.997158 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:37 UTC)" (scheduled at 2025-02-07 03:44:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:44:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:44:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:48.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 03:44:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:48 UTC)" (scheduled at 2025-02-07 03:44:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:44:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:44:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 38, 732925), True, datetime.datetime(2025, 2, 7, 3, 44, 38, 732925), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:44:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:44:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:44:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:44:38,933 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:44:39,457 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:44:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:44:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:54.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 03:44:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:10 UTC)" (scheduled at 2025-02-07 03:44:40.424250+00:00)
gcworker stdout | 2025-02-07 03:44:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:44:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899580431, None, 1, 0])
gcworker stdout | 2025-02-07 03:44:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:44:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:44:40,629 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:44:41,501 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:44:41,629 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:44:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:44:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:11.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 03:44:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:11 UTC)" (scheduled at 2025-02-07 03:44:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:44:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896281836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:44:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:44:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:44:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:44:42,678 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:42,979 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:44:43,650 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:44:43,774 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:44:43,807 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:44:44,135 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:44:44,235 [238] [DEBUG] [app] Starting request: urn:request:9b8768e6-33a5-4a4e-b6d8-7738a2f0da1b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:44,235 [235] [DEBUG] [app] Starting request: urn:request:02d171b7-4a84-4a19-8259-9d3318f410bd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:44:44,238 [239] [DEBUG] [app] Starting request: urn:request:933efcfb-e950-4124-aa53-53f157758027 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:44,239 [239] [DEBUG] [app] Ending request: urn:request:933efcfb-e950-4124-aa53-53f157758027 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:933efcfb-e950-4124-aa53-53f157758027', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:44:44,239 [246] [DEBUG] [app] Starting request: urn:request:6e89746d-4edf-4508-a9d7-f1dc768c0f67 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:44:44,239 [246] [DEBUG] [app] Ending request: urn:request:6e89746d-4edf-4508-a9d7-f1dc768c0f67 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e89746d-4edf-4508-a9d7-f1dc768c0f67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:44:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:44,241 [237] [DEBUG] [app] Starting request: urn:request:ba2980af-3df8-4016-ae41-040e442e19bb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:44,242 [237] [DEBUG] [app] Ending request: urn:request:ba2980af-3df8-4016-ae41-040e442e19bb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba2980af-3df8-4016-ae41-040e442e19bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:44:44,242 [235] [DEBUG] [app] Starting request: urn:request:27d0691c-6e3c-458f-9017-63b873f2836a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:44,243 [235] [DEBUG] [app] Ending request: urn:request:27d0691c-6e3c-458f-9017-63b873f2836a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27d0691c-6e3c-458f-9017-63b873f2836a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:44:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:44,255 [235] [DEBUG] [app] Ending request: urn:request:02d171b7-4a84-4a19-8259-9d3318f410bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:02d171b7-4a84-4a19-8259-9d3318f410bd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:44,255 [238] [DEBUG] [app] Ending request: urn:request:9b8768e6-33a5-4a4e-b6d8-7738a2f0da1b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b8768e6-33a5-4a4e-b6d8-7738a2f0da1b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:44:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
gunicorn-web stdout | 2025-02-07 03:44:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:44:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gcworker stdout | 2025-02-07 03:44:44,666 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:44:44,746 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:44:44,793 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:44:46,013 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:44:46,017 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:44:46,021 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:44:46,024 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:44:46,026 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:44:46,457 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:44:46,734 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:44:46,976 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:44:47,045 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:44:47,048 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:44:47,050 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:44:48,028 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:44:48,031 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:44:48,034 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:44:48,037 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:44:48,042 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:44:48,046 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:44:48,048 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:44:48,052 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:44:48,056 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:44:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:44:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:50.814677+00:00 (in 2.082091 seconds)
notificationworker stdout | 2025-02-07 03:44:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:58 UTC)" (scheduled at 2025-02-07 03:44:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:44:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:44:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 48, 732861), True, datetime.datetime(2025, 2, 7, 3, 44, 48, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:44:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:44:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:44:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:44:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:44:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:44:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:58.732147+00:00 (in 7.917046 seconds)
notificationworker stdout | 2025-02-07 03:44:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:50 UTC)" (scheduled at 2025-02-07 03:44:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:44:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:44:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:44:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:44:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:44:59.937899+00:00 (in 6.999773 seconds)
exportactionlogsworker stdout | 2025-02-07 03:44:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:49:52 UTC)" (scheduled at 2025-02-07 03:44:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:44:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 44, 52, 938396), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:44:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 44, 52, 938396), True, datetime.datetime(2025, 2, 7, 3, 44, 52, 938396), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:44:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 44, 52, 938396), True, datetime.datetime(2025, 2, 7, 3, 44, 52, 938396), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 44, 52, 938396), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:44:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:44:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:49:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:44:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:44:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:10.424250+00:00 (in 16.000698 seconds)
gcworker stdout | 2025-02-07 03:44:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:24 UTC)" (scheduled at 2025-02-07 03:44:54.423055+00:00)
gcworker stdout | 2025-02-07 03:44:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:44:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:44:55,982 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:44:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:44:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:56.118855+00:00 (in 59.999500 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:44:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:56 UTC)" (scheduled at 2025-02-07 03:44:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:44:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:44:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:44:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:37.715878+00:00 (in 41.001866 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:56 UTC)" (scheduled at 2025-02-07 03:44:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:44:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:44:57,955 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:44:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:44:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:08.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 03:44:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:08 UTC)" (scheduled at 2025-02-07 03:44:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:44:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:44:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 44, 58, 732883), True, datetime.datetime(2025, 2, 7, 3, 44, 58, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:44:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:44:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:44:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:44:58,988 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:44:59,235 [236] [DEBUG] [app] Starting request: urn:request:cb43759e-52ce-4b3e-9dd2-ddb59fe41858 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:59,235 [237] [DEBUG] [app] Starting request: urn:request:1584e9fb-d18c-4ae9-88f9-a04c2a584a9e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:44:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:44:59,238 [246] [DEBUG] [app] Starting request: urn:request:2452c397-fe64-4483-95ac-9047018ebab6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:59,239 [244] [DEBUG] [app] Starting request: urn:request:379352b7-88b9-4fd0-a3cc-ca66f2e95580 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:44:59,239 [246] [DEBUG] [app] Ending request: urn:request:2452c397-fe64-4483-95ac-9047018ebab6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2452c397-fe64-4483-95ac-9047018ebab6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:44:59,239 [244] [DEBUG] [app] Ending request: urn:request:379352b7-88b9-4fd0-a3cc-ca66f2e95580 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:379352b7-88b9-4fd0-a3cc-ca66f2e95580', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:44:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:44:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:44:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:44:59,243 [235] [DEBUG] [app] Starting request: urn:request:75548e73-1c42-4877-8f79-8dd4e0f56cbd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:59,243 [238] [DEBUG] [app] Starting request: urn:request:e51d8a57-3805-4d38-ba3f-a06f5a585c9c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:44:59,243 [235] [DEBUG] [app] Ending request: urn:request:75548e73-1c42-4877-8f79-8dd4e0f56cbd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:75548e73-1c42-4877-8f79-8dd4e0f56cbd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:44:59,243 [238] [DEBUG] [app] Ending request: urn:request:e51d8a57-3805-4d38-ba3f-a06f5a585c9c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e51d8a57-3805-4d38-ba3f-a06f5a585c9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:44:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:44:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:44:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:44:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:44:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:44:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:44:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:44:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:44:59,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:44:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:44:59,255 [237] [DEBUG] [app] Ending request: urn:request:1584e9fb-d18c-4ae9-88f9-a04c2a584a9e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1584e9fb-d18c-4ae9-88f9-a04c2a584a9e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:59,255 [236] [DEBUG] [app] Ending request: urn:request:cb43759e-52ce-4b3e-9dd2-ddb59fe41858 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb43759e-52ce-4b3e-9dd2-ddb59fe41858', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:44:59,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:44:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:44:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:44:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:44:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:44:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
exportactionlogsworker stdout | 2025-02-07 03:44:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:44:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:04.936688+00:00 (in 4.998312 seconds)
exportactionlogsworker stdout | 2025-02-07 03:44:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:59 UTC)" (scheduled at 2025-02-07 03:44:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:44:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:44:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:45:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:45:03,306 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:45:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:45:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:23.420886+00:00 (in 20.001301 seconds)
repositorygcworker stdout | 2025-02-07 03:45:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:03 UTC)" (scheduled at 2025-02-07 03:45:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:45:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:45:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 3, 419871), True, datetime.datetime(2025, 2, 7, 3, 45, 3, 419871), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:45:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:45:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:45:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:45:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:45:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:33.919286+00:00 (in 29.999474 seconds)
buildlogsarchiver stdout | 2025-02-07 03:45:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:33 UTC)" (scheduled at 2025-02-07 03:45:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:45:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 45, 3, 920079), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:45:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:45:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:45:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:45:04,311 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:45:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:45:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:59.937899+00:00 (in 55.000749 seconds)
exportactionlogsworker stdout | 2025-02-07 03:45:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:04 UTC)" (scheduled at 2025-02-07 03:45:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:45:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:45:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 4, 937466), True, datetime.datetime(2025, 2, 7, 3, 45, 4, 937466), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:45:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:45:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:45:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:45:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:45:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:18.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 03:45:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:18 UTC)" (scheduled at 2025-02-07 03:45:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:45:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:45:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 8, 732810), True, datetime.datetime(2025, 2, 7, 3, 45, 8, 732810), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:45:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:45:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:45:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:45:08,969 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:45:09,477 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:45:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:45:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:24.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 03:45:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:40 UTC)" (scheduled at 2025-02-07 03:45:10.424250+00:00)
gcworker stdout | 2025-02-07 03:45:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:45:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899610431, None, 1, 0])
gcworker stdout | 2025-02-07 03:45:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:45:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:45:10,659 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:45:11,534 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:45:11,640 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:45:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:45:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:41.831521+00:00 (in 29.999548 seconds)
autopruneworker stdout | 2025-02-07 03:45:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:41 UTC)" (scheduled at 2025-02-07 03:45:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:45:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896311836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:45:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:45:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:45:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:45:12,714 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:13,015 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:45:13,686 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:45:13,810 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
globalpromstats stdout | 2025-02-07 03:45:13,818 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
repositorygcworker stdout | 2025-02-07 03:45:14,172 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:45:14,236 [235] [DEBUG] [app] Starting request: urn:request:3a4e6ea9-11a2-4d35-99f2-8b28140afd7d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:14,236 [238] [DEBUG] [app] Starting request: urn:request:4fc13b09-31fb-4f27-91d5-dcff179b1d83 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:45:14,239 [239] [DEBUG] [app] Starting request: urn:request:8eb25b47-b8d3-4f2b-bb12-fc78b2f1f8bd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:45:14,239 [246] [DEBUG] [app] Starting request: urn:request:15af4f53-ec08-4068-b5ac-1b729605b445 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:45:14,239 [239] [DEBUG] [app] Ending request: urn:request:8eb25b47-b8d3-4f2b-bb12-fc78b2f1f8bd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8eb25b47-b8d3-4f2b-bb12-fc78b2f1f8bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:45:14,239 [246] [DEBUG] [app] Ending request: urn:request:15af4f53-ec08-4068-b5ac-1b729605b445 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15af4f53-ec08-4068-b5ac-1b729605b445', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:45:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:45:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:45:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:14,243 [235] [DEBUG] [app] Starting request: urn:request:06625c8f-b688-4923-ab9a-69519127ecb3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:14,243 [235] [DEBUG] [app] Ending request: urn:request:06625c8f-b688-4923-ab9a-69519127ecb3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06625c8f-b688-4923-ab9a-69519127ecb3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:14,243 [236] [DEBUG] [app] Starting request: urn:request:01c0ebb6-0d57-448e-885a-541554599a7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:45:14,244 [236] [DEBUG] [app] Ending request: urn:request:01c0ebb6-0d57-448e-885a-541554599a7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01c0ebb6-0d57-448e-885a-541554599a7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:45:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:14,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:14,255 [235] [DEBUG] [app] Ending request: urn:request:3a4e6ea9-11a2-4d35-99f2-8b28140afd7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a4e6ea9-11a2-4d35-99f2-8b28140afd7d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:45:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:45:14,256 [238] [DEBUG] [app] Ending request: urn:request:4fc13b09-31fb-4f27-91d5-dcff179b1d83 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4fc13b09-31fb-4f27-91d5-dcff179b1d83', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:45:14,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gcworker stdout | 2025-02-07 03:45:14,703 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:45:14,769 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:45:14,821 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:45:16,021 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:45:16,026 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:45:16,030 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:45:16,032 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:45:16,035 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:45:16,493 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:45:16,756 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:45:17,001 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:45:17,053 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:45:17,056 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:45:17,059 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:45:18,037 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:45:18,040 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:45:18,045 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:45:18,047 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:45:18,054 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:45:18,057 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:45:18,060 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:45:18,062 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:45:18,065 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:45:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:45:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:28.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 03:45:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:28 UTC)" (scheduled at 2025-02-07 03:45:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:45:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:45:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 18, 732813), True, datetime.datetime(2025, 2, 7, 3, 45, 18, 732813), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:45:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:45:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:45:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:45:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:45:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:03.419110+00:00 (in 39.997748 seconds)
repositorygcworker stdout | 2025-02-07 03:45:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:23 UTC)" (scheduled at 2025-02-07 03:45:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:45:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:45:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:45:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:45:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:40.424250+00:00 (in 16.000731 seconds)
gcworker stdout | 2025-02-07 03:45:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:54 UTC)" (scheduled at 2025-02-07 03:45:24.423055+00:00)
gcworker stdout | 2025-02-07 03:45:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:45:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:45:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:45:26,011 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:45:27,991 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:45:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:45:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:38.732147+00:00 (in 9.999534 seconds)
notificationworker stdout | 2025-02-07 03:45:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:38 UTC)" (scheduled at 2025-02-07 03:45:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:45:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:45:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 28, 732823), True, datetime.datetime(2025, 2, 7, 3, 45, 28, 732823), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:45:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:45:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:45:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:45:28,999 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:45:29,236 [235] [DEBUG] [app] Starting request: urn:request:07f7de06-0b5c-4ba6-9833-0707775f2ad4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:29,236 [237] [DEBUG] [app] Starting request: urn:request:b35e6f52-dd92-4b30-8d69-038699d78940 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:45:29,239 [246] [DEBUG] [app] Starting request: urn:request:29c11b14-4bc9-4fff-b343-9bc4311b0f0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:45:29,239 [239] [DEBUG] [app] Starting request: urn:request:03ccb769-6f02-496e-b19c-ca9cd9aa15cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:45:29,239 [246] [DEBUG] [app] Ending request: urn:request:29c11b14-4bc9-4fff-b343-9bc4311b0f0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:29c11b14-4bc9-4fff-b343-9bc4311b0f0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:45:29,239 [239] [DEBUG] [app] Ending request: urn:request:03ccb769-6f02-496e-b19c-ca9cd9aa15cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03ccb769-6f02-496e-b19c-ca9cd9aa15cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:45:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:45:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:45:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:29,242 [236] [DEBUG] [app] Starting request: urn:request:e3c9e788-daa8-4a52-b175-7f84b8a1d1e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:29,242 [237] [DEBUG] [app] Starting request: urn:request:ce022ced-f1b1-429a-914c-86f037a00cc5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:29,242 [236] [DEBUG] [app] Ending request: urn:request:e3c9e788-daa8-4a52-b175-7f84b8a1d1e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e3c9e788-daa8-4a52-b175-7f84b8a1d1e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:29,242 [237] [DEBUG] [app] Ending request: urn:request:ce022ced-f1b1-429a-914c-86f037a00cc5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ce022ced-f1b1-429a-914c-86f037a00cc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:45:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:45:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:29,254 [237] [DEBUG] [app] Ending request: urn:request:b35e6f52-dd92-4b30-8d69-038699d78940 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b35e6f52-dd92-4b30-8d69-038699d78940', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:29,254 [235] [DEBUG] [app] Ending request: urn:request:07f7de06-0b5c-4ba6-9833-0707775f2ad4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07f7de06-0b5c-4ba6-9833-0707775f2ad4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:45:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:45:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:45:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 03:45:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:45:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:32.625945+00:00 (in 1.004292 seconds)
namespacegcworker stdout | 2025-02-07 03:45:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:31 UTC)" (scheduled at 2025-02-07 03:45:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:45:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:45:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 31, 621982), True, datetime.datetime(2025, 2, 7, 3, 45, 31, 621982), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:45:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:45:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:45:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:45:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:45:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:31.621131+00:00 (in 58.994593 seconds)
namespacegcworker stdout | 2025-02-07 03:45:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:32 UTC)" (scheduled at 2025-02-07 03:45:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:45:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:45:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:45:33,333 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:45:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:45:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:03.919286+00:00 (in 29.999375 seconds)
buildlogsarchiver stdout | 2025-02-07 03:45:33,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:03 UTC)" (scheduled at 2025-02-07 03:45:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:45:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 45, 33, 920311), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:45:33,928 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:45:33,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:45:33,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:45:34,332 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:56.713547+00:00 (in 18.997193 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:37 UTC)" (scheduled at 2025-02-07 03:45:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:45:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:45:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:48.732147+00:00 (in 9.999517 seconds)
notificationworker stdout | 2025-02-07 03:45:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:48 UTC)" (scheduled at 2025-02-07 03:45:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:45:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:45:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 38, 732844), True, datetime.datetime(2025, 2, 7, 3, 45, 38, 732844), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:45:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:45:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:45:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:45:38,976 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:45:39,493 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:45:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:45:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:54.423055+00:00 (in 13.998350 seconds)
gcworker stdout | 2025-02-07 03:45:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:10 UTC)" (scheduled at 2025-02-07 03:45:40.424250+00:00)
gcworker stdout | 2025-02-07 03:45:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:45:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899640431, None, 1, 0])
gcworker stdout | 2025-02-07 03:45:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:45:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:45:40,696 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:45:41,570 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:45:41,677 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:45:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:45:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:11.831521+00:00 (in 29.999527 seconds)
autopruneworker stdout | 2025-02-07 03:45:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:11 UTC)" (scheduled at 2025-02-07 03:45:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:45:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896341836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:45:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:45:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:45:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:45:42,750 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:43,051 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:45:43,717 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:45:43,839 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:45:43,846 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:45:44,207 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:45:44,235 [238] [DEBUG] [app] Starting request: urn:request:0ec6842a-46d1-4b9d-8c07-abd2a2e1a766 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:44,235 [235] [DEBUG] [app] Starting request: urn:request:2a5eba7b-236d-479c-b5c5-aeefef19ea41 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:45:44,238 [246] [DEBUG] [app] Starting request: urn:request:46dae65f-4b2e-4760-b656-081f4de62c0e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:45:44,239 [246] [DEBUG] [app] Ending request: urn:request:46dae65f-4b2e-4760-b656-081f4de62c0e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:46dae65f-4b2e-4760-b656-081f4de62c0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:45:44,239 [239] [DEBUG] [app] Starting request: urn:request:4f00b072-c549-408d-b33f-873f3a4a8104 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:45:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:45:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:45:44,239 [239] [DEBUG] [app] Ending request: urn:request:4f00b072-c549-408d-b33f-873f3a4a8104 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f00b072-c549-408d-b33f-873f3a4a8104', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:45:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:45:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:44,241 [237] [DEBUG] [app] Starting request: urn:request:8fc62d82-48a5-4fa5-9b19-6d53ebd32f8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:44,242 [237] [DEBUG] [app] Ending request: urn:request:8fc62d82-48a5-4fa5-9b19-6d53ebd32f8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8fc62d82-48a5-4fa5-9b19-6d53ebd32f8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:45:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:45:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:44,243 [237] [DEBUG] [app] Starting request: urn:request:c45d567f-a83e-409d-805a-66a4de833972 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:44,243 [237] [DEBUG] [app] Ending request: urn:request:c45d567f-a83e-409d-805a-66a4de833972 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c45d567f-a83e-409d-805a-66a4de833972', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:45:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:44,253 [235] [DEBUG] [app] Ending request: urn:request:2a5eba7b-236d-479c-b5c5-aeefef19ea41 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a5eba7b-236d-479c-b5c5-aeefef19ea41', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:45:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 03:45:44,254 [238] [DEBUG] [app] Ending request: urn:request:0ec6842a-46d1-4b9d-8c07-abd2a2e1a766 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ec6842a-46d1-4b9d-8c07-abd2a2e1a766', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:45:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gcworker stdout | 2025-02-07 03:45:44,730 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:45:44,805 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:45:44,843 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:45:46,029 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:45:46,033 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:45:46,036 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:45:46,039 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:45:46,042 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:45:46,511 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:45:46,769 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:45:47,018 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:45:47,060 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:45:47,063 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:45:47,066 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:45:48,047 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:45:48,052 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:45:48,056 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:45:48,059 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:45:48,066 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:45:48,071 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:45:48,074 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:45:48,077 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:45:48,079 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:45:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:45:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:50.814677+00:00 (in 2.082060 seconds)
notificationworker stdout | 2025-02-07 03:45:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:58 UTC)" (scheduled at 2025-02-07 03:45:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:45:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:45:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 48, 732828), True, datetime.datetime(2025, 2, 7, 3, 45, 48, 732828), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:45:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:45:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:45:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:45:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:45:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:45:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:45:58.732147+00:00 (in 7.916989 seconds)
notificationworker stdout | 2025-02-07 03:45:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:50 UTC)" (scheduled at 2025-02-07 03:45:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:45:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:45:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:45:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:45:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:10.424250+00:00 (in 16.000731 seconds)
gcworker stdout | 2025-02-07 03:45:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:24 UTC)" (scheduled at 2025-02-07 03:45:54.423055+00:00)
gcworker stdout | 2025-02-07 03:45:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:45:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:45:56,048 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:45:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:45:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:56.118855+00:00 (in 59.999525 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:45:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:56 UTC)" (scheduled at 2025-02-07 03:45:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:45:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:45:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:45:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:37.715878+00:00 (in 41.001861 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:56 UTC)" (scheduled at 2025-02-07 03:45:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:45:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:45:58,028 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:45:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:45:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:08.732147+00:00 (in 9.999529 seconds)
notificationworker stdout | 2025-02-07 03:45:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:08 UTC)" (scheduled at 2025-02-07 03:45:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:45:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:45:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 45, 58, 732825), True, datetime.datetime(2025, 2, 7, 3, 45, 58, 732825), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:45:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:45:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:45:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:45:59,035 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:45:59,235 [236] [DEBUG] [app] Starting request: urn:request:2bd52c16-7543-4812-a1e6-8e86d270c0fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:59,235 [235] [DEBUG] [app] Starting request: urn:request:03e7c616-9f3e-4270-ae13-980bdeeb9458 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:45:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:59,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:45:59,239 [246] [DEBUG] [app] Starting request: urn:request:8d0da552-809f-4a3c-81e8-3172c9744c2b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:45:59,239 [245] [DEBUG] [app] Starting request: urn:request:04e264f6-882c-4799-b783-098d284314d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:45:59,240 [246] [DEBUG] [app] Ending request: urn:request:8d0da552-809f-4a3c-81e8-3172c9744c2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d0da552-809f-4a3c-81e8-3172c9744c2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:45:59,240 [245] [DEBUG] [app] Ending request: urn:request:04e264f6-882c-4799-b783-098d284314d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04e264f6-882c-4799-b783-098d284314d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:45:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:45:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:45:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:59,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:59,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:45:59,244 [238] [DEBUG] [app] Starting request: urn:request:e287c80c-7450-4ed9-88f6-73ec7ab583ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:59,244 [237] [DEBUG] [app] Starting request: urn:request:b9c03f74-afbc-4280-92b5-30b1e7193a17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:45:59,244 [237] [DEBUG] [app] Ending request: urn:request:b9c03f74-afbc-4280-92b5-30b1e7193a17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9c03f74-afbc-4280-92b5-30b1e7193a17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:59,244 [238] [DEBUG] [app] Ending request: urn:request:e287c80c-7450-4ed9-88f6-73ec7ab583ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e287c80c-7450-4ed9-88f6-73ec7ab583ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:45:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:45:59,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:45:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:45:59,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:45:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:45:59,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:45:59,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:59,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:45:59,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:59,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:45:59,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:59,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:45:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:45:59,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:59,254 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:45:59,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:59,255 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:45:59,256 [235] [DEBUG] [app] Ending request: urn:request:03e7c616-9f3e-4270-ae13-980bdeeb9458 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:03e7c616-9f3e-4270-ae13-980bdeeb9458', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:45:59,256 [236] [DEBUG] [app] Ending request: urn:request:2bd52c16-7543-4812-a1e6-8e86d270c0fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2bd52c16-7543-4812-a1e6-8e86d270c0fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:45:59,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:45:59,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:45:59,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:45:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:45:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.022)
exportactionlogsworker stdout | 2025-02-07 03:45:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:45:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:04.936688+00:00 (in 4.998293 seconds)
exportactionlogsworker stdout | 2025-02-07 03:45:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:59 UTC)" (scheduled at 2025-02-07 03:45:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:45:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:45:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:46:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:46:03,354 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:46:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:46:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:23.420886+00:00 (in 20.001287 seconds)
repositorygcworker stdout | 2025-02-07 03:46:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:03 UTC)" (scheduled at 2025-02-07 03:46:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:46:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:46:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 3, 419883), True, datetime.datetime(2025, 2, 7, 3, 46, 3, 419883), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:46:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:46:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:46:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:46:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:46:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:33.919286+00:00 (in 29.999495 seconds)
buildlogsarchiver stdout | 2025-02-07 03:46:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:33 UTC)" (scheduled at 2025-02-07 03:46:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:46:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 46, 3, 920045), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:46:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:46:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:46:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:46:04,351 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:46:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:46:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:59.937899+00:00 (in 55.000750 seconds)
exportactionlogsworker stdout | 2025-02-07 03:46:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:04 UTC)" (scheduled at 2025-02-07 03:46:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:46:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:46:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 4, 937462), True, datetime.datetime(2025, 2, 7, 3, 46, 4, 937462), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:46:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:46:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:46:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:46:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:46:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:18.732147+00:00 (in 9.999529 seconds)
notificationworker stdout | 2025-02-07 03:46:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:18 UTC)" (scheduled at 2025-02-07 03:46:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:46:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:46:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 8, 732830), True, datetime.datetime(2025, 2, 7, 3, 46, 8, 732830), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:46:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:46:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:46:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:46:09,012 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 03:46:09,223 [77] [DEBUG] [workers.worker] Scheduling worker.
quotatotalworker stdout | 2025-02-07 03:46:09,224 [77] [INFO] [apscheduler.scheduler] Scheduler started
quotatotalworker stdout | 2025-02-07 03:46:09,224 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:46:09,224 [77] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
quotatotalworker stdout | 2025-02-07 03:46:09,224 [77] [DEBUG] [workers.worker] First run scheduled for 2025-02-07 03:46:14.225378
quotatotalworker stdout | 2025-02-07 03:46:09,224 [77] [INFO] [apscheduler.scheduler] Added job "QuotaTotalWorker.backfill" to job store "default"
quotatotalworker stdout | 2025-02-07 03:46:09,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:46:09,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:14.225378+00:00 (in 5.000218 seconds)
logrotateworker stdout | 2025-02-07 03:46:09,529 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:46:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:46:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:24.423055+00:00 (in 13.998372 seconds)
gcworker stdout | 2025-02-07 03:46:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:40 UTC)" (scheduled at 2025-02-07 03:46:10.424250+00:00)
gcworker stdout | 2025-02-07 03:46:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:46:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899670431, None, 1, 0])
gcworker stdout | 2025-02-07 03:46:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:46:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:46:10,730 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:46:11,606 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:46:11,711 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:46:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:46:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:41.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 03:46:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:41 UTC)" (scheduled at 2025-02-07 03:46:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:46:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896371836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:46:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:46:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:46:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:46:12,759 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:13,080 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:46:13,754 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:46:13,849 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:46:13,870 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:46:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:46:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:46:14 UTC)" (scheduled at 2025-02-07 03:46:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:46:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:46:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:46:14,227 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:29.225378+00:00 (in 14.997667 seconds)
quotatotalworker stdout | 2025-02-07 03:46:14,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:46:14,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:46:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:46:14,235 [235] [DEBUG] [app] Starting request: urn:request:e348d217-ce4c-445c-9bdf-0e5d22e2a6df (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:14,235 [237] [DEBUG] [app] Starting request: urn:request:635b77ac-6039-43ec-b7b6-2316c23a0ec7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:46:14,238 [245] [DEBUG] [app] Starting request: urn:request:3f97682d-4c71-41f3-8ed1-d505ab1baeb4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:14,238 [239] [DEBUG] [app] Starting request: urn:request:294279f1-6e03-46ab-84d2-45f41c7c8c99 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:14,239 [245] [DEBUG] [app] Ending request: urn:request:3f97682d-4c71-41f3-8ed1-d505ab1baeb4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f97682d-4c71-41f3-8ed1-d505ab1baeb4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:46:14,239 [239] [DEBUG] [app] Ending request: urn:request:294279f1-6e03-46ab-84d2-45f41c7c8c99 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:294279f1-6e03-46ab-84d2-45f41c7c8c99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:46:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:46:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:46:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:14,242 [235] [DEBUG] [app] Starting request: urn:request:91e68b72-1f8a-4cee-a823-4bec30c7d7d0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:14,242 [238] [DEBUG] [app] Starting request: urn:request:5de3be0b-8208-4ad7-94e2-fc1a4e82deb2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:14,242 [235] [DEBUG] [app] Ending request: urn:request:91e68b72-1f8a-4cee-a823-4bec30c7d7d0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91e68b72-1f8a-4cee-a823-4bec30c7d7d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:14,243 [238] [DEBUG] [app] Ending request: urn:request:5de3be0b-8208-4ad7-94e2-fc1a4e82deb2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5de3be0b-8208-4ad7-94e2-fc1a4e82deb2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:46:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:46:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:46:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
repositorygcworker stdout | 2025-02-07 03:46:14,244 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 03:46:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:14,254 [235] [DEBUG] [app] Ending request: urn:request:e348d217-ce4c-445c-9bdf-0e5d22e2a6df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e348d217-ce4c-445c-9bdf-0e5d22e2a6df', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:46:14,254 [237] [DEBUG] [app] Ending request: urn:request:635b77ac-6039-43ec-b7b6-2316c23a0ec7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:635b77ac-6039-43ec-b7b6-2316c23a0ec7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:46:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:46:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gcworker stdout | 2025-02-07 03:46:14,767 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:46:14,818 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:46:14,880 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:46:16,038 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:46:16,041 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:46:16,044 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:46:16,046 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:46:16,050 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:46:16,529 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:46:16,805 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:46:17,025 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:46:17,067 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:46:17,070 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:46:17,073 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:46:18,061 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:46:18,064 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:46:18,068 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:46:18,071 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:46:18,076 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:46:18,081 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:46:18,085 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:46:18,089 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:46:18,092 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:46:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:46:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:28.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 03:46:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:28 UTC)" (scheduled at 2025-02-07 03:46:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:46:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:46:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 18, 732815), True, datetime.datetime(2025, 2, 7, 3, 46, 18, 732815), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:46:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:46:18,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:46:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:46:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:46:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:03.419110+00:00 (in 39.997741 seconds)
repositorygcworker stdout | 2025-02-07 03:46:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:23 UTC)" (scheduled at 2025-02-07 03:46:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:46:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:46:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:46:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:46:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:40.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 03:46:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:54 UTC)" (scheduled at 2025-02-07 03:46:24.423055+00:00)
gcworker stdout | 2025-02-07 03:46:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:46:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:46:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:46:26,084 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:46:28,041 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:46:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:46:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:38.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 03:46:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:38 UTC)" (scheduled at 2025-02-07 03:46:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:46:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:46:28,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 28, 732943), True, datetime.datetime(2025, 2, 7, 3, 46, 28, 732943), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:46:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:46:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:46:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:46:29,053 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
quotatotalworker stdout | 2025-02-07 03:46:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:46:29,226 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:44.225378+00:00 (in 14.999346 seconds)
quotatotalworker stdout | 2025-02-07 03:46:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:46:44 UTC)" (scheduled at 2025-02-07 03:46:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:46:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:46:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:46:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:46:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:46:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:46:29,235 [235] [DEBUG] [app] Starting request: urn:request:aa6c3005-a93d-4f9e-8f5c-742e2cbde3cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:29,235 [238] [DEBUG] [app] Starting request: urn:request:8ae8559b-7a1d-42b7-aaf5-a721517ddca6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:46:29,239 [246] [DEBUG] [app] Starting request: urn:request:ad70e7bd-457f-4fbc-96ea-9876c5b8e391 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:29,239 [239] [DEBUG] [app] Starting request: urn:request:78283277-fd57-4f7d-94ba-cf86bdccfbc4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:29,239 [246] [DEBUG] [app] Ending request: urn:request:ad70e7bd-457f-4fbc-96ea-9876c5b8e391 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ad70e7bd-457f-4fbc-96ea-9876c5b8e391', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:46:29,239 [239] [DEBUG] [app] Ending request: urn:request:78283277-fd57-4f7d-94ba-cf86bdccfbc4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78283277-fd57-4f7d-94ba-cf86bdccfbc4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:46:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:46:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:46:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:29,242 [238] [DEBUG] [app] Starting request: urn:request:9535dce7-c8d1-4c88-b926-5254ec59df84 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:29,242 [238] [DEBUG] [app] Ending request: urn:request:9535dce7-c8d1-4c88-b926-5254ec59df84 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9535dce7-c8d1-4c88-b926-5254ec59df84', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:46:29,243 [237] [DEBUG] [app] Starting request: urn:request:fc9a2a95-37c9-4ad0-bb72-e629ceed1454 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:29,243 [237] [DEBUG] [app] Ending request: urn:request:fc9a2a95-37c9-4ad0-bb72-e629ceed1454 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc9a2a95-37c9-4ad0-bb72-e629ceed1454', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:46:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:46:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:46:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:29,254 [238] [DEBUG] [app] Ending request: urn:request:8ae8559b-7a1d-42b7-aaf5-a721517ddca6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ae8559b-7a1d-42b7-aaf5-a721517ddca6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:29,254 [235] [DEBUG] [app] Ending request: urn:request:aa6c3005-a93d-4f9e-8f5c-742e2cbde3cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aa6c3005-a93d-4f9e-8f5c-742e2cbde3cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:46:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:46:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:46:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:46:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:46:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:32.625945+00:00 (in 1.004318 seconds)
namespacegcworker stdout | 2025-02-07 03:46:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:31 UTC)" (scheduled at 2025-02-07 03:46:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:46:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:46:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 31, 621905), True, datetime.datetime(2025, 2, 7, 3, 46, 31, 621905), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:46:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:46:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:46:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:46:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:46:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:13.624301+00:00 (in 40.997888 seconds)
namespacegcworker stdout | 2025-02-07 03:46:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:32 UTC)" (scheduled at 2025-02-07 03:46:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:46:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:46:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:46:33,390 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:46:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:46:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:03.919286+00:00 (in 29.999529 seconds)
buildlogsarchiver stdout | 2025-02-07 03:46:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:03 UTC)" (scheduled at 2025-02-07 03:46:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:46:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 46, 33, 920023), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:46:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:46:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:46:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:46:34,360 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:56.713547+00:00 (in 18.997163 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:37 UTC)" (scheduled at 2025-02-07 03:46:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:37 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:46:38,468 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:46:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:46:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:48.732147+00:00 (in 9.999531 seconds)
notificationworker stdout | 2025-02-07 03:46:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:48 UTC)" (scheduled at 2025-02-07 03:46:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:46:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:46:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 38, 732950), True, datetime.datetime(2025, 2, 7, 3, 46, 38, 732950), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:46:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:46:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:46:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:46:39,020 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:46:39,566 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:46:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:46:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:54.423055+00:00 (in 13.998350 seconds)
gcworker stdout | 2025-02-07 03:46:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:10 UTC)" (scheduled at 2025-02-07 03:46:40.424250+00:00)
gcworker stdout | 2025-02-07 03:46:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:46:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899700431, None, 1, 0])
gcworker stdout | 2025-02-07 03:46:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:46:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:46:40,767 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:46:41,643 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:46:41,747 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:46:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:46:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:11.831521+00:00 (in 29.999545 seconds)
autopruneworker stdout | 2025-02-07 03:46:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:11 UTC)" (scheduled at 2025-02-07 03:46:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:46:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896401836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:46:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:46:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:46:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:46:42,765 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:43,117 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:46:43,791 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:46:43,883 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:46:43,906 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:46:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:46:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:59.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 03:46:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:46:59 UTC)" (scheduled at 2025-02-07 03:46:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:46:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:46:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:46:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:46:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:46:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:46:44,236 [238] [DEBUG] [app] Starting request: urn:request:df8cba06-4a9e-492d-8085-75b20e808f51 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:44,236 [237] [DEBUG] [app] Starting request: urn:request:e747e25f-9166-4221-943e-ab5ef5ba16d8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:46:44,239 [239] [DEBUG] [app] Starting request: urn:request:f1841520-c14f-4646-b316-f9c6cd7a75cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:44,239 [245] [DEBUG] [app] Starting request: urn:request:c3feeb77-bcf1-4583-bb13-c6ee40ac9900 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:44,239 [239] [DEBUG] [app] Ending request: urn:request:f1841520-c14f-4646-b316-f9c6cd7a75cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f1841520-c14f-4646-b316-f9c6cd7a75cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:46:44,239 [245] [DEBUG] [app] Ending request: urn:request:c3feeb77-bcf1-4583-bb13-c6ee40ac9900 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3feeb77-bcf1-4583-bb13-c6ee40ac9900', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:46:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:46:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:46:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:44,243 [238] [DEBUG] [app] Starting request: urn:request:e0f67087-7b1c-43bf-99f3-89ef3c86ae73 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:44,243 [238] [DEBUG] [app] Ending request: urn:request:e0f67087-7b1c-43bf-99f3-89ef3c86ae73 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0f67087-7b1c-43bf-99f3-89ef3c86ae73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:44,243 [235] [DEBUG] [app] Starting request: urn:request:8545a40f-385a-4c01-806a-2e2b08bf99cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:46:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:44,244 [235] [DEBUG] [app] Ending request: urn:request:8545a40f-385a-4c01-806a-2e2b08bf99cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8545a40f-385a-4c01-806a-2e2b08bf99cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:46:44,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:46:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:46:44,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:46:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:44,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:44,255 [238] [DEBUG] [app] Ending request: urn:request:df8cba06-4a9e-492d-8085-75b20e808f51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df8cba06-4a9e-492d-8085-75b20e808f51', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:46:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:46:44,256 [237] [DEBUG] [app] Ending request: urn:request:e747e25f-9166-4221-943e-ab5ef5ba16d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e747e25f-9166-4221-943e-ab5ef5ba16d8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:46:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
repositorygcworker stdout | 2025-02-07 03:46:44,280 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:46:44,786 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:46:44,854 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:46:44,916 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:46:46,047 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:46:46,049 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:46:46,053 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:46:46,056 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:46:46,058 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:46:46,566 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:46:46,831 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:46:47,034 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:46:47,074 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:46:47,078 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:46:47,081 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:46:48,072 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:46:48,075 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:46:48,077 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:46:48,080 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:46:48,086 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:46:48,089 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:46:48,092 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:46:48,097 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:46:48,100 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:46:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:46:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:50.814677+00:00 (in 2.082079 seconds)
notificationworker stdout | 2025-02-07 03:46:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:58 UTC)" (scheduled at 2025-02-07 03:46:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:46:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:46:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 48, 732806), True, datetime.datetime(2025, 2, 7, 3, 46, 48, 732806), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:46:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:46:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:46:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:46:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:46:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:46:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:46:58.732147+00:00 (in 7.916994 seconds)
notificationworker stdout | 2025-02-07 03:46:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:50 UTC)" (scheduled at 2025-02-07 03:46:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:46:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:46:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:46:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:46:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:10.424250+00:00 (in 16.000706 seconds)
gcworker stdout | 2025-02-07 03:46:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:24 UTC)" (scheduled at 2025-02-07 03:46:54.423055+00:00)
gcworker stdout | 2025-02-07 03:46:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:46:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:46:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:46:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:56.118855+00:00 (in 59.999548 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:46:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:56 UTC)" (scheduled at 2025-02-07 03:46:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:46:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
storagereplication stdout | 2025-02-07 03:46:56,121 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotaregistrysizeworker stdout | 2025-02-07 03:46:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:46:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:37.715878+00:00 (in 41.001843 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:56 UTC)" (scheduled at 2025-02-07 03:46:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:46:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:46:58,055 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:46:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:46:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:08.732147+00:00 (in 9.999494 seconds)
notificationworker stdout | 2025-02-07 03:46:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:08 UTC)" (scheduled at 2025-02-07 03:46:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:46:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:46:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 46, 58, 732865), True, datetime.datetime(2025, 2, 7, 3, 46, 58, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:46:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:46:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:46:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:46:59,060 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
quotatotalworker stdout | 2025-02-07 03:46:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:46:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:14.225378+00:00 (in 14.999530 seconds)
quotatotalworker stdout | 2025-02-07 03:46:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:14 UTC)" (scheduled at 2025-02-07 03:46:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:46:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:46:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:46:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:46:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:46:59,235 [235] [DEBUG] [app] Starting request: urn:request:33ab7f39-05e6-44f8-960c-fa083eccaa6e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:59,235 [238] [DEBUG] [app] Starting request: urn:request:9bc45439-50eb-4657-9d3d-42069bbfd621 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:46:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:46:59,238 [239] [DEBUG] [app] Starting request: urn:request:f9ebcbab-3e2f-454b-b55a-83cb09908da4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:59,239 [246] [DEBUG] [app] Starting request: urn:request:2d4413bb-f458-46a2-9a5d-2f995a1f4783 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:46:59,239 [239] [DEBUG] [app] Ending request: urn:request:f9ebcbab-3e2f-454b-b55a-83cb09908da4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9ebcbab-3e2f-454b-b55a-83cb09908da4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:46:59,239 [246] [DEBUG] [app] Ending request: urn:request:2d4413bb-f458-46a2-9a5d-2f995a1f4783 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d4413bb-f458-46a2-9a5d-2f995a1f4783', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:46:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:46:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:46:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:46:59,242 [238] [DEBUG] [app] Starting request: urn:request:8ecbba76-7f9b-427d-b8bf-e0c2db3dd8c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:59,242 [238] [DEBUG] [app] Ending request: urn:request:8ecbba76-7f9b-427d-b8bf-e0c2db3dd8c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ecbba76-7f9b-427d-b8bf-e0c2db3dd8c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:46:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:59,243 [236] [DEBUG] [app] Starting request: urn:request:a567f75c-5676-4d03-8fc8-7996a38300ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:46:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:46:59,243 [236] [DEBUG] [app] Ending request: urn:request:a567f75c-5676-4d03-8fc8-7996a38300ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a567f75c-5676-4d03-8fc8-7996a38300ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:46:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:46:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:46:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:46:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:46:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:46:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:46:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:46:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:46:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:46:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:46:59,254 [238] [DEBUG] [app] Ending request: urn:request:9bc45439-50eb-4657-9d3d-42069bbfd621 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9bc45439-50eb-4657-9d3d-42069bbfd621', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:46:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 03:46:59,255 [235] [DEBUG] [app] Ending request: urn:request:33ab7f39-05e6-44f8-960c-fa083eccaa6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33ab7f39-05e6-44f8-960c-fa083eccaa6e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:46:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:46:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:46:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:46:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 03:46:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:46:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:04.936688+00:00 (in 4.998275 seconds)
exportactionlogsworker stdout | 2025-02-07 03:46:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:59 UTC)" (scheduled at 2025-02-07 03:46:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:46:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:46:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:47:59 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:47:03,418 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
repositorygcworker stdout | 2025-02-07 03:47:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:47:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:23.420886+00:00 (in 20.001387 seconds)
repositorygcworker stdout | 2025-02-07 03:47:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:03 UTC)" (scheduled at 2025-02-07 03:47:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:47:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:47:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 3, 419810), True, datetime.datetime(2025, 2, 7, 3, 47, 3, 419810), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:47:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:47:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:47:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:47:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:47:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:33.919286+00:00 (in 29.999510 seconds)
buildlogsarchiver stdout | 2025-02-07 03:47:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:33 UTC)" (scheduled at 2025-02-07 03:47:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:47:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 47, 3, 920047), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:47:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:47:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:47:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:47:04,396 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:47:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:47:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:59.937899+00:00 (in 55.000770 seconds)
exportactionlogsworker stdout | 2025-02-07 03:47:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:04 UTC)" (scheduled at 2025-02-07 03:47:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:47:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:47:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 4, 937440), True, datetime.datetime(2025, 2, 7, 3, 47, 4, 937440), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:47:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:47:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:47:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:04 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:47:08,505 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:47:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:18.732147+00:00 (in 9.999507 seconds)
notificationworker stdout | 2025-02-07 03:47:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:18 UTC)" (scheduled at 2025-02-07 03:47:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:47:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:47:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 8, 732854), True, datetime.datetime(2025, 2, 7, 3, 47, 8, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:47:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:47:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:47:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:47:09,057 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:47:09,602 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:47:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:47:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:24.423055+00:00 (in 13.998323 seconds)
gcworker stdout | 2025-02-07 03:47:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:40 UTC)" (scheduled at 2025-02-07 03:47:10.424250+00:00)
gcworker stdout | 2025-02-07 03:47:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:47:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899730431, None, 1, 0])
gcworker stdout | 2025-02-07 03:47:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:47:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:47:10,774 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:47:11,679 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:47:11,782 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:47:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:47:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:41.831521+00:00 (in 29.999538 seconds)
autopruneworker stdout | 2025-02-07 03:47:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:41 UTC)" (scheduled at 2025-02-07 03:47:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:47:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896431836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:47:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:47:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:47:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:41 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:47:12,775 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:13,152 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
namespacegcworker stdout | 2025-02-07 03:47:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:47:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:31.621131+00:00 (in 17.996366 seconds)
namespacegcworker stdout | 2025-02-07 03:47:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:52:13 UTC)" (scheduled at 2025-02-07 03:47:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:47:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 47, 13, 625025), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:47:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 47, 13, 625025), True, datetime.datetime(2025, 2, 7, 3, 47, 13, 625025), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:47:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 47, 13, 625025), True, datetime.datetime(2025, 2, 7, 3, 47, 13, 625025), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 47, 13, 625025), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:47:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:47:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:52:13 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:47:13,828 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:47:13,918 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:47:13,920 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:47:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:47:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:29.225378+00:00 (in 14.999534 seconds)
quotatotalworker stdout | 2025-02-07 03:47:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:29 UTC)" (scheduled at 2025-02-07 03:47:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:47:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:47:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:47:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:47:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:47:14,236 [235] [DEBUG] [app] Starting request: urn:request:7e466cf7-6e8c-454e-812b-4eac4b326637 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:14,236 [237] [DEBUG] [app] Starting request: urn:request:941be61d-e540-4422-a18d-b2864be27e3e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:14,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:47:14,239 [243] [DEBUG] [app] Starting request: urn:request:8c9622b5-3ca1-4825-bc61-d7fe387e8072 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:14,239 [245] [DEBUG] [app] Starting request: urn:request:d66415ac-e5ca-460a-a691-36318c569355 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:14,240 [243] [DEBUG] [app] Ending request: urn:request:8c9622b5-3ca1-4825-bc61-d7fe387e8072 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c9622b5-3ca1-4825-bc61-d7fe387e8072', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:47:14,240 [245] [DEBUG] [app] Ending request: urn:request:d66415ac-e5ca-460a-a691-36318c569355 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d66415ac-e5ca-460a-a691-36318c569355', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:47:14,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:47:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:47:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:14,242 [237] [DEBUG] [app] Starting request: urn:request:c1361285-d4af-468a-9e00-868da33bfd5f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:14,243 [237] [DEBUG] [app] Ending request: urn:request:c1361285-d4af-468a-9e00-868da33bfd5f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c1361285-d4af-468a-9e00-868da33bfd5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:47:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:47:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:14,243 [236] [DEBUG] [app] Starting request: urn:request:3e71025b-3bac-4fe6-a75d-3e805eeeaf40 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:14,244 [236] [DEBUG] [app] Ending request: urn:request:3e71025b-3bac-4fe6-a75d-3e805eeeaf40 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e71025b-3bac-4fe6-a75d-3e805eeeaf40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:47:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:14,254 [237] [DEBUG] [app] Ending request: urn:request:941be61d-e540-4422-a18d-b2864be27e3e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:941be61d-e540-4422-a18d-b2864be27e3e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:47:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:47:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:47:14,255 [235] [DEBUG] [app] Ending request: urn:request:7e466cf7-6e8c-454e-812b-4eac4b326637 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e466cf7-6e8c-454e-812b-4eac4b326637', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:47:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:47:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
repositorygcworker stdout | 2025-02-07 03:47:14,316 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:47:14,811 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:47:14,890 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:47:14,949 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:47:16,054 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:47:16,057 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:47:16,061 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:47:16,064 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:47:16,067 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:47:16,585 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:47:16,842 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:47:17,071 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-secscan stdout | 2025-02-07 03:47:17,082 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:47:17,086 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:47:17,089 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-registry stdout | 2025-02-07 03:47:18,083 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:47:18,086 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:47:18,089 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:47:18,092 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:47:18,095 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:47:18,097 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:47:18,100 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:47:18,105 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:47:18,109 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:47:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:28.732147+00:00 (in 9.999518 seconds)
notificationworker stdout | 2025-02-07 03:47:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:28 UTC)" (scheduled at 2025-02-07 03:47:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:47:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:47:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 18, 732952), True, datetime.datetime(2025, 2, 7, 3, 47, 18, 732952), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:47:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:47:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:47:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:47:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:47:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:03.419110+00:00 (in 39.997738 seconds)
repositorygcworker stdout | 2025-02-07 03:47:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:23 UTC)" (scheduled at 2025-02-07 03:47:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:47:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:47:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:47:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:47:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 03:47:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:54 UTC)" (scheduled at 2025-02-07 03:47:24.423055+00:00)
gcworker stdout | 2025-02-07 03:47:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:47:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:47:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:47:26,129 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:47:28,078 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:47:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:33.814166+00:00 (in 5.081475 seconds)
notificationworker stdout | 2025-02-07 03:47:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:38 UTC)" (scheduled at 2025-02-07 03:47:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:47:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:47:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 28, 733021), True, datetime.datetime(2025, 2, 7, 3, 47, 28, 733021), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:47:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:47:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:47:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:47:29,096 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
quotatotalworker stdout | 2025-02-07 03:47:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:47:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:44.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 03:47:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:44 UTC)" (scheduled at 2025-02-07 03:47:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:47:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:47:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:47:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:47:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:47:29,236 [235] [DEBUG] [app] Starting request: urn:request:2732814e-272c-4dda-85cd-f76b1d9b48ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:29,236 [238] [DEBUG] [app] Starting request: urn:request:fcf99f78-a7f0-4d0b-b166-1aa23588f83f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:47:29,239 [239] [DEBUG] [app] Starting request: urn:request:be8ed9af-3040-48d0-9dec-c6277f8aa043 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:29,239 [245] [DEBUG] [app] Starting request: urn:request:ff6bb31f-77b1-4d66-8e34-2e2377ccc19e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:29,239 [239] [DEBUG] [app] Ending request: urn:request:be8ed9af-3040-48d0-9dec-c6277f8aa043 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be8ed9af-3040-48d0-9dec-c6277f8aa043', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:47:29,239 [245] [DEBUG] [app] Ending request: urn:request:ff6bb31f-77b1-4d66-8e34-2e2377ccc19e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff6bb31f-77b1-4d66-8e34-2e2377ccc19e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:47:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:47:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:47:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:29,242 [235] [DEBUG] [app] Starting request: urn:request:8f2dd680-5c43-45e0-b7fd-125995fee758 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:29,242 [235] [DEBUG] [app] Ending request: urn:request:8f2dd680-5c43-45e0-b7fd-125995fee758 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8f2dd680-5c43-45e0-b7fd-125995fee758', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:47:29,242 [237] [DEBUG] [app] Starting request: urn:request:415fc1c2-7235-43c2-9d9f-91b15c76401f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:29,243 [237] [DEBUG] [app] Ending request: urn:request:415fc1c2-7235-43c2-9d9f-91b15c76401f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:415fc1c2-7235-43c2-9d9f-91b15c76401f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:47:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:47:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:29,254 [235] [DEBUG] [app] Ending request: urn:request:2732814e-272c-4dda-85cd-f76b1d9b48ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2732814e-272c-4dda-85cd-f76b1d9b48ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:47:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:47:29,254 [238] [DEBUG] [app] Ending request: urn:request:fcf99f78-a7f0-4d0b-b166-1aa23588f83f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fcf99f78-a7f0-4d0b-b166-1aa23588f83f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:47:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:47:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:47:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 03:47:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:47:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:32.625945+00:00 (in 1.004343 seconds)
namespacegcworker stdout | 2025-02-07 03:47:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:31 UTC)" (scheduled at 2025-02-07 03:47:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:47:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:47:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 31, 621892), True, datetime.datetime(2025, 2, 7, 3, 47, 31, 621892), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:47:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:47:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:47:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:47:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:47:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:31.621131+00:00 (in 58.994731 seconds)
namespacegcworker stdout | 2025-02-07 03:47:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:32 UTC)" (scheduled at 2025-02-07 03:47:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:47:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:47:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:47:33,444 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 03:47:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:38.732147+00:00 (in 4.917472 seconds)
notificationworker stdout | 2025-02-07 03:47:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:52:33 UTC)" (scheduled at 2025-02-07 03:47:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:47:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 47, 33, 814860), 'notification/%'])
notificationworker stdout | 2025-02-07 03:47:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 47, 33, 814860), True, datetime.datetime(2025, 2, 7, 3, 47, 33, 814860), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:47:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 47, 33, 814860), True, datetime.datetime(2025, 2, 7, 3, 47, 33, 814860), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 47, 33, 814860), 'notification/%'])
notificationworker stdout | 2025-02-07 03:47:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:47:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:52:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:47:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:47:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:03.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 03:47:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:03 UTC)" (scheduled at 2025-02-07 03:47:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:47:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 47, 33, 920016), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:47:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:47:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:47:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:47:34,432 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:56.713547+00:00 (in 18.997183 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:37 UTC)" (scheduled at 2025-02-07 03:47:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:37 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:47:38,531 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:47:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:48.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 03:47:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:48 UTC)" (scheduled at 2025-02-07 03:47:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:47:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:47:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 38, 732878), True, datetime.datetime(2025, 2, 7, 3, 47, 38, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:47:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:47:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:47:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:47:39,090 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:47:39,639 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:47:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:47:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:54.423055+00:00 (in 13.998235 seconds)
gcworker stdout | 2025-02-07 03:47:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:10 UTC)" (scheduled at 2025-02-07 03:47:40.424250+00:00)
gcworker stdout | 2025-02-07 03:47:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:47:40,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899760431, None, 1, 0])
gcworker stdout | 2025-02-07 03:47:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:47:40,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:47:40,794 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:47:41,717 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:47:41,818 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:47:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:47:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:11.831521+00:00 (in 29.999542 seconds)
autopruneworker stdout | 2025-02-07 03:47:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:11 UTC)" (scheduled at 2025-02-07 03:47:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:47:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896461836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:47:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:47:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:47:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:11 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 03:47:42,811 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:43,164 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:47:43,853 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:47:43,935 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:47:43,957 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:47:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:47:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:59.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 03:47:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:59 UTC)" (scheduled at 2025-02-07 03:47:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:47:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:47:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:47:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:47:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:47:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:47:44,235 [238] [DEBUG] [app] Starting request: urn:request:abb5cd11-3256-479f-aaba-ca46257f63df (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:44,235 [236] [DEBUG] [app] Starting request: urn:request:b39e6ea2-6b1b-4d0c-8b4f-66e15582cc68 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:47:44,238 [239] [DEBUG] [app] Starting request: urn:request:4a6cdd0a-8ca2-491b-9ac6-a11ef33950fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:44,238 [246] [DEBUG] [app] Starting request: urn:request:95ec0206-a58f-4ff4-b532-601a22ceb1da (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:44,238 [239] [DEBUG] [app] Ending request: urn:request:4a6cdd0a-8ca2-491b-9ac6-a11ef33950fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a6cdd0a-8ca2-491b-9ac6-a11ef33950fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:47:44,238 [246] [DEBUG] [app] Ending request: urn:request:95ec0206-a58f-4ff4-b532-601a22ceb1da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:95ec0206-a58f-4ff4-b532-601a22ceb1da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:47:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:47:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:47:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:44,242 [235] [DEBUG] [app] Starting request: urn:request:97d16516-c546-4156-ab04-a5874dbafeba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:44,242 [237] [DEBUG] [app] Starting request: urn:request:92dba04b-3579-4a7d-b78d-474432fe918d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:44,242 [237] [DEBUG] [app] Ending request: urn:request:92dba04b-3579-4a7d-b78d-474432fe918d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92dba04b-3579-4a7d-b78d-474432fe918d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:47:44,242 [235] [DEBUG] [app] Ending request: urn:request:97d16516-c546-4156-ab04-a5874dbafeba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97d16516-c546-4156-ab04-a5874dbafeba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:47:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:44,254 [236] [DEBUG] [app] Ending request: urn:request:b39e6ea2-6b1b-4d0c-8b4f-66e15582cc68 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b39e6ea2-6b1b-4d0c-8b4f-66e15582cc68', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:47:44,254 [238] [DEBUG] [app] Ending request: urn:request:abb5cd11-3256-479f-aaba-ca46257f63df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:abb5cd11-3256-479f-aaba-ca46257f63df', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:47:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:47:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:47:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:47:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositorygcworker stdout | 2025-02-07 03:47:44,353 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:47:44,847 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:47:44,915 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:47:44,965 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:47:46,063 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:47:46,067 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:47:46,070 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:47:46,073 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:47:46,076 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:47:46,621 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:47:46,879 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:47:47,090 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:47:47,095 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:47:47,098 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:47:47,107 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:47:48,095 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:47:48,097 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:47:48,100 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:47:48,103 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:47:48,107 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:47:48,110 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:47:48,113 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:47:48,115 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:47:48,119 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:47:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:50.814677+00:00 (in 2.082092 seconds)
notificationworker stdout | 2025-02-07 03:47:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:58 UTC)" (scheduled at 2025-02-07 03:47:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:47:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:47:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 48, 732863), True, datetime.datetime(2025, 2, 7, 3, 47, 48, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:47:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:47:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:47:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:47:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:47:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:47:58.732147+00:00 (in 7.917014 seconds)
notificationworker stdout | 2025-02-07 03:47:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:50 UTC)" (scheduled at 2025-02-07 03:47:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:47:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:47:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:47:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:47:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:10.424250+00:00 (in 16.000671 seconds)
gcworker stdout | 2025-02-07 03:47:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:24 UTC)" (scheduled at 2025-02-07 03:47:54.423055+00:00)
gcworker stdout | 2025-02-07 03:47:54,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 03:47:54,430 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:47:54,430 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:47:54,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:47:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:47:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:56.118855+00:00 (in 59.999513 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:47:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:56 UTC)" (scheduled at 2025-02-07 03:47:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:47:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:47:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:47:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:47:56,158 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:37.715878+00:00 (in 41.001863 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:56 UTC)" (scheduled at 2025-02-07 03:47:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:47:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:47:58,114 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:47:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:47:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:08.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 03:47:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:08 UTC)" (scheduled at 2025-02-07 03:47:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:47:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:47:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 47, 58, 732868), True, datetime.datetime(2025, 2, 7, 3, 47, 58, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:47:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:47:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:47:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:47:59,132 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
quotatotalworker stdout | 2025-02-07 03:47:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:47:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:14.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 03:47:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:14 UTC)" (scheduled at 2025-02-07 03:47:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:47:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:47:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:47:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:47:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:47:59,236 [237] [DEBUG] [app] Starting request: urn:request:ed861adf-8c1d-4788-900f-c4af3d9eddbd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:59,236 [235] [DEBUG] [app] Starting request: urn:request:01e7f57d-415f-48f7-b184-0f83d6707015 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:47:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:47:59,239 [244] [DEBUG] [app] Starting request: urn:request:d92b8dae-5a99-484e-9a33-fa5d470f7e02 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:59,239 [239] [DEBUG] [app] Starting request: urn:request:d98b83cb-22d7-46b5-bec4-9b784dbd8662 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:47:59,239 [244] [DEBUG] [app] Ending request: urn:request:d92b8dae-5a99-484e-9a33-fa5d470f7e02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d92b8dae-5a99-484e-9a33-fa5d470f7e02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:47:59,239 [239] [DEBUG] [app] Ending request: urn:request:d98b83cb-22d7-46b5-bec4-9b784dbd8662 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d98b83cb-22d7-46b5-bec4-9b784dbd8662', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:47:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:47:59,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:47:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:59,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:47:59,242 [235] [DEBUG] [app] Starting request: urn:request:f7a6a761-16c0-4037-860e-6b75a623662a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:59,242 [235] [DEBUG] [app] Ending request: urn:request:f7a6a761-16c0-4037-860e-6b75a623662a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7a6a761-16c0-4037-860e-6b75a623662a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:47:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:47:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:59,243 [236] [DEBUG] [app] Starting request: urn:request:0d13ce21-203b-4c56-a863-2cf59714b742 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:47:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:59,243 [236] [DEBUG] [app] Ending request: urn:request:0d13ce21-203b-4c56-a863-2cf59714b742 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d13ce21-203b-4c56-a863-2cf59714b742', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:47:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:47:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:47:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:47:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:47:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:47:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:47:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:47:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:47:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:47:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:47:59,254 [235] [DEBUG] [app] Ending request: urn:request:01e7f57d-415f-48f7-b184-0f83d6707015 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01e7f57d-415f-48f7-b184-0f83d6707015', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:47:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:47:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:47:59,254 [237] [DEBUG] [app] Ending request: urn:request:ed861adf-8c1d-4788-900f-c4af3d9eddbd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed861adf-8c1d-4788-900f-c4af3d9eddbd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:47:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:47:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:47:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:47:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:47:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:47:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:04.936688+00:00 (in 4.998323 seconds)
exportactionlogsworker stdout | 2025-02-07 03:47:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:59 UTC)" (scheduled at 2025-02-07 03:47:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:47:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:47:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:48:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:48:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:48:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:23.420886+00:00 (in 20.001337 seconds)
repositorygcworker stdout | 2025-02-07 03:48:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:03 UTC)" (scheduled at 2025-02-07 03:48:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:48:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:48:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 3, 419822), True, datetime.datetime(2025, 2, 7, 3, 48, 3, 419822), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:48:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:48:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:48:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:48:03,480 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:48:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:48:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:33.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 03:48:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:33 UTC)" (scheduled at 2025-02-07 03:48:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:48:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 48, 3, 919987), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:48:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:48:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:48:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:48:04,465 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:48:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:48:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:59.937899+00:00 (in 55.000775 seconds)
exportactionlogsworker stdout | 2025-02-07 03:48:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:04 UTC)" (scheduled at 2025-02-07 03:48:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:48:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:48:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 4, 937421), True, datetime.datetime(2025, 2, 7, 3, 48, 4, 937421), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:48:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:48:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:48:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:04 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:48:08,554 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:48:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:48:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:18.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 03:48:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:18 UTC)" (scheduled at 2025-02-07 03:48:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:48:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:48:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 8, 732852), True, datetime.datetime(2025, 2, 7, 3, 48, 8, 732852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:48:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:48:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:48:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:48:09,108 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:48:09,652 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:48:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:48:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:24.423055+00:00 (in 13.998309 seconds)
gcworker stdout | 2025-02-07 03:48:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:40 UTC)" (scheduled at 2025-02-07 03:48:10.424250+00:00)
gcworker stdout | 2025-02-07 03:48:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:48:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899790431, None, 1, 0])
gcworker stdout | 2025-02-07 03:48:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:48:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:48:10,818 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:48:11,748 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
autopruneworker stdout | 2025-02-07 03:48:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:48:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:41.831521+00:00 (in 29.999552 seconds)
autopruneworker stdout | 2025-02-07 03:48:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:41 UTC)" (scheduled at 2025-02-07 03:48:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:48:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896491836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:48:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:48:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:48:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:41 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 03:48:11,855 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:48:12,842 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:13,200 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:48:13,890 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:48:13,947 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:48:13,993 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:48:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:48:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:29.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 03:48:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:29 UTC)" (scheduled at 2025-02-07 03:48:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:48:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:48:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:48:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:48:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:48:14,235 [236] [DEBUG] [app] Starting request: urn:request:62ac993e-a1ab-4484-82d6-a32898a19da5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:14,235 [235] [DEBUG] [app] Starting request: urn:request:3462e082-d433-4736-a077-dd51088c91d3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:48:14,238 [239] [DEBUG] [app] Starting request: urn:request:ecb00b75-ec4e-4eca-ab05-ee0d9bc4ea37 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:14,238 [246] [DEBUG] [app] Starting request: urn:request:028b709b-6fd7-499f-ac9b-7512aca8f044 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:14,239 [239] [DEBUG] [app] Ending request: urn:request:ecb00b75-ec4e-4eca-ab05-ee0d9bc4ea37 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ecb00b75-ec4e-4eca-ab05-ee0d9bc4ea37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:48:14,239 [246] [DEBUG] [app] Ending request: urn:request:028b709b-6fd7-499f-ac9b-7512aca8f044 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:028b709b-6fd7-499f-ac9b-7512aca8f044', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:48:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:48:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:48:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:14,242 [237] [DEBUG] [app] Starting request: urn:request:8d8298a2-08e3-423b-b7f6-2f1a86e5d02a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:48:14,242 [237] [DEBUG] [app] Ending request: urn:request:8d8298a2-08e3-423b-b7f6-2f1a86e5d02a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d8298a2-08e3-423b-b7f6-2f1a86e5d02a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:14,242 [236] [DEBUG] [app] Starting request: urn:request:c1a2df9a-0cfd-4654-bf00-ed6486d80589 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:48:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:48:14,242 [236] [DEBUG] [app] Ending request: urn:request:c1a2df9a-0cfd-4654-bf00-ed6486d80589 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c1a2df9a-0cfd-4654-bf00-ed6486d80589', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:48:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:48:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:48:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:14,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:14,255 [236] [DEBUG] [app] Ending request: urn:request:62ac993e-a1ab-4484-82d6-a32898a19da5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62ac993e-a1ab-4484-82d6-a32898a19da5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:48:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:48:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:48:14,255 [235] [DEBUG] [app] Ending request: urn:request:3462e082-d433-4736-a077-dd51088c91d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3462e082-d433-4736-a077-dd51088c91d3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:48:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:48:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
repositorygcworker stdout | 2025-02-07 03:48:14,390 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:48:14,859 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:48:14,932 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:48:14,983 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:48:16,071 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:48:16,074 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:48:16,079 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:48:16,082 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:48:16,085 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:48:16,634 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:48:16,896 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:48:17,100 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:48:17,104 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:48:17,108 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:48:17,140 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:48:18,105 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:48:18,108 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:48:18,111 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:48:18,114 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:48:18,118 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:48:18,121 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:48:18,124 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:48:18,126 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:48:18,129 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:48:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:48:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:28.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 03:48:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:28 UTC)" (scheduled at 2025-02-07 03:48:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:48:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:48:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 18, 732913), True, datetime.datetime(2025, 2, 7, 3, 48, 18, 732913), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:48:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:48:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:48:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:48:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:48:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:32.420707+00:00 (in 8.999316 seconds)
repositorygcworker stdout | 2025-02-07 03:48:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:23 UTC)" (scheduled at 2025-02-07 03:48:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:48:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:48:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:48:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:48:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:40.424250+00:00 (in 16.000728 seconds)
gcworker stdout | 2025-02-07 03:48:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:54 UTC)" (scheduled at 2025-02-07 03:48:24.423055+00:00)
gcworker stdout | 2025-02-07 03:48:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:48:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:48:54 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 03:48:24,735 [63] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2025-02-07 03:48:24,736 [63] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-02-07 03:48:24 UTC)" (scheduled at 2025-02-07 03:48:24.735445+00:00)
globalpromstats stdout | 2025-02-07 03:48:24,736 [63] [DEBUG] [__main__] Attempting to report stats
globalpromstats stdout | 2025-02-07 03:48:24,736 [63] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 03:48:24,736 [63] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ...
globalpromstats stdout | 2025-02-07 03:48:24,736 [63] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:24.735445+00:00 (in 3599.998845 seconds)
globalpromstats stdout | 2025-02-07 03:48:24,739 [63] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2025-02-07 03:48:24,739 [63] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 03:48:24,739 [63] [DEBUG] [__main__] Reporting global stats
globalpromstats stdout | 2025-02-07 03:48:24,740 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2025-02-07 03:48:24,746 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False])
globalpromstats stdout | 2025-02-07 03:48:24,748 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True])
globalpromstats stdout | 2025-02-07 03:48:24,750 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2025-02-07 03:48:24,751 [63] [DEBUG] [data.database] Disconnecting from database.
globalpromstats stdout | 2025-02-07 03:48:24,751 [63] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 03:48:24,751 [63] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2025-02-07 03:48:24,752 [63] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 03:48:24,752 [63] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-02-07 04:48:24 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:48:26,169 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:48:28,151 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:48:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:48:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:38.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 03:48:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:38 UTC)" (scheduled at 2025-02-07 03:48:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:48:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:48:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 28, 732906), True, datetime.datetime(2025, 2, 7, 3, 48, 28, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:48:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:48:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:48:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:38 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:48:29,162 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
quotatotalworker stdout | 2025-02-07 03:48:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:48:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:44.225378+00:00 (in 14.999505 seconds)
quotatotalworker stdout | 2025-02-07 03:48:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:44 UTC)" (scheduled at 2025-02-07 03:48:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:48:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:48:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:48:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:48:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:48:29,236 [235] [DEBUG] [app] Starting request: urn:request:5474f597-cb03-475f-bb64-0324b830a65a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:29,236 [237] [DEBUG] [app] Starting request: urn:request:3017b385-2bf4-4217-a6ba-c0ca3cc7f600 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:29,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:48:29,239 [246] [DEBUG] [app] Starting request: urn:request:546af662-cf3a-4a63-b622-dbb2e9dfcbba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:29,239 [246] [DEBUG] [app] Ending request: urn:request:546af662-cf3a-4a63-b622-dbb2e9dfcbba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:546af662-cf3a-4a63-b622-dbb2e9dfcbba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:48:29,239 [245] [DEBUG] [app] Starting request: urn:request:fcfd9f50-5589-4189-b46d-499abdeb772c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:29,240 [245] [DEBUG] [app] Ending request: urn:request:fcfd9f50-5589-4189-b46d-499abdeb772c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fcfd9f50-5589-4189-b46d-499abdeb772c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:48:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:48:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:48:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:29,242 [237] [DEBUG] [app] Starting request: urn:request:270d5939-094e-4fb1-984a-3ee5c7589709 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:48:29,243 [237] [DEBUG] [app] Ending request: urn:request:270d5939-094e-4fb1-984a-3ee5c7589709 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:270d5939-094e-4fb1-984a-3ee5c7589709', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:29,243 [238] [DEBUG] [app] Starting request: urn:request:26e94cbe-89f9-4b89-84b8-2e4084358cd4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:48:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:48:29,243 [238] [DEBUG] [app] Ending request: urn:request:26e94cbe-89f9-4b89-84b8-2e4084358cd4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26e94cbe-89f9-4b89-84b8-2e4084358cd4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:48:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:48:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:29,255 [235] [DEBUG] [app] Ending request: urn:request:5474f597-cb03-475f-bb64-0324b830a65a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5474f597-cb03-475f-bb64-0324b830a65a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:48:29,255 [237] [DEBUG] [app] Ending request: urn:request:3017b385-2bf4-4217-a6ba-c0ca3cc7f600 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3017b385-2bf4-4217-a6ba-c0ca3cc7f600', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:48:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:48:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:48:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:48:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 03:48:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:48:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:32.625945+00:00 (in 1.004324 seconds)
namespacegcworker stdout | 2025-02-07 03:48:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:31 UTC)" (scheduled at 2025-02-07 03:48:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:48:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:48:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 31, 621944), True, datetime.datetime(2025, 2, 7, 3, 48, 31, 621944), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:48:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:48:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:48:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:48:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:48:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:03.419110+00:00 (in 30.997938 seconds)
repositorygcworker stdout | 2025-02-07 03:48:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:53:32 UTC)" (scheduled at 2025-02-07 03:48:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:48:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 48, 32, 421456), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:48:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 48, 32, 421456), True, datetime.datetime(2025, 2, 7, 3, 48, 32, 421456), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:48:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 48, 32, 421456), True, datetime.datetime(2025, 2, 7, 3, 48, 32, 421456), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 48, 32, 421456), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:48:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:48:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:53:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:48:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:48:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:31.621131+00:00 (in 58.994625 seconds)
namespacegcworker stdout | 2025-02-07 03:48:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:32 UTC)" (scheduled at 2025-02-07 03:48:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:48:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:48:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:48:33,505 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:48:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:48:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:03.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 03:48:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:03 UTC)" (scheduled at 2025-02-07 03:48:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:48:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 48, 33, 919991), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:48:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:48:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:48:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:48:34,474 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:56.713547+00:00 (in 18.997221 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:37 UTC)" (scheduled at 2025-02-07 03:48:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:37 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:48:38,578 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:48:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:48:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:48.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 03:48:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:48 UTC)" (scheduled at 2025-02-07 03:48:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:48:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:48:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 38, 732864), True, datetime.datetime(2025, 2, 7, 3, 48, 38, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:48:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:48:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:48:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:48:39,145 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:48:39,671 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:48:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:48:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:54.423055+00:00 (in 13.998367 seconds)
gcworker stdout | 2025-02-07 03:48:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:10 UTC)" (scheduled at 2025-02-07 03:48:40.424250+00:00)
gcworker stdout | 2025-02-07 03:48:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:48:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899820431, None, 1, 0])
gcworker stdout | 2025-02-07 03:48:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:48:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:48:40,841 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:48:41,779 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
autopruneworker stdout | 2025-02-07 03:48:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:48:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:11.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 03:48:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:11 UTC)" (scheduled at 2025-02-07 03:48:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:48:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896521836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:48:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:48:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:48:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:11 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 03:48:41,891 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:48:42,879 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:43,231 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:48:43,926 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:48:43,971 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:48:44,022 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:48:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:48:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:59.225378+00:00 (in 14.999479 seconds)
quotatotalworker stdout | 2025-02-07 03:48:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:59 UTC)" (scheduled at 2025-02-07 03:48:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:48:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:48:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:48:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:48:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:48:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:48:44,236 [237] [DEBUG] [app] Starting request: urn:request:c0f3a123-fcad-404b-9050-855e63902f42 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:44,236 [235] [DEBUG] [app] Starting request: urn:request:ad2d304b-fda7-4fe4-a521-8a0ef30594ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:48:44,239 [246] [DEBUG] [app] Starting request: urn:request:e9cc9d63-0c35-419e-b58f-37902c8ff29f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:44,239 [239] [DEBUG] [app] Starting request: urn:request:e88571f0-24d8-4143-8232-66ca7725cca5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:44,239 [239] [DEBUG] [app] Ending request: urn:request:e88571f0-24d8-4143-8232-66ca7725cca5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e88571f0-24d8-4143-8232-66ca7725cca5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:48:44,239 [246] [DEBUG] [app] Ending request: urn:request:e9cc9d63-0c35-419e-b58f-37902c8ff29f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e9cc9d63-0c35-419e-b58f-37902c8ff29f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:48:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:48:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:48:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:44,242 [235] [DEBUG] [app] Starting request: urn:request:0100f3f0-5c11-4d8b-9bbe-f2de28d20d4f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:48:44,242 [238] [DEBUG] [app] Starting request: urn:request:9ab7d449-ea31-4594-bdbe-4211f5bb53fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:48:44,242 [235] [DEBUG] [app] Ending request: urn:request:0100f3f0-5c11-4d8b-9bbe-f2de28d20d4f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0100f3f0-5c11-4d8b-9bbe-f2de28d20d4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:44,242 [238] [DEBUG] [app] Ending request: urn:request:9ab7d449-ea31-4594-bdbe-4211f5bb53fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ab7d449-ea31-4594-bdbe-4211f5bb53fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:48:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:48:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:48:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:44,254 [237] [DEBUG] [app] Ending request: urn:request:c0f3a123-fcad-404b-9050-855e63902f42 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c0f3a123-fcad-404b-9050-855e63902f42', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:48:44,254 [235] [DEBUG] [app] Ending request: urn:request:ad2d304b-fda7-4fe4-a521-8a0ef30594ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ad2d304b-fda7-4fe4-a521-8a0ef30594ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:48:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:48:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:48:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:48:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositorygcworker stdout | 2025-02-07 03:48:44,424 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:48:44,888 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:48:44,954 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:48:45,019 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:48:46,080 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:48:46,084 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:48:46,086 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:48:46,090 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:48:46,093 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:48:46,658 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:48:46,932 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:48:47,109 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:48:47,114 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:48:47,117 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:48:47,162 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:48:48,115 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:48:48,119 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:48:48,122 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:48:48,124 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:48:48,127 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:48:48,131 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:48:48,134 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:48:48,138 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:48:48,140 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:48:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:48:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:50.814677+00:00 (in 2.082085 seconds)
notificationworker stdout | 2025-02-07 03:48:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:58 UTC)" (scheduled at 2025-02-07 03:48:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:48:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:48:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 48, 732883), True, datetime.datetime(2025, 2, 7, 3, 48, 48, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:48:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:48:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:48:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:48:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:48:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:48:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:48:58.732147+00:00 (in 7.917018 seconds)
notificationworker stdout | 2025-02-07 03:48:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:50 UTC)" (scheduled at 2025-02-07 03:48:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:48:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:48:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:48:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:48:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:10.424250+00:00 (in 16.000701 seconds)
gcworker stdout | 2025-02-07 03:48:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:24 UTC)" (scheduled at 2025-02-07 03:48:54.423055+00:00)
gcworker stdout | 2025-02-07 03:48:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:48:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:48:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:48:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:56.118855+00:00 (in 59.999518 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:48:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:56 UTC)" (scheduled at 2025-02-07 03:48:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:48:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:48:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:48:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:48:56,205 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:37.715878+00:00 (in 41.001857 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:56 UTC)" (scheduled at 2025-02-07 03:48:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:56,715 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:48:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:48:58,171 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:48:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:48:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:08.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 03:48:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:08 UTC)" (scheduled at 2025-02-07 03:48:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:48:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:48:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 48, 58, 732912), True, datetime.datetime(2025, 2, 7, 3, 48, 58, 732912), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:48:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:48:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:48:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:08 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:48:59,198 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
quotatotalworker stdout | 2025-02-07 03:48:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:48:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:14.225378+00:00 (in 14.999524 seconds)
quotatotalworker stdout | 2025-02-07 03:48:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:14 UTC)" (scheduled at 2025-02-07 03:48:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:48:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:48:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:48:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:48:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:48:59,235 [235] [DEBUG] [app] Starting request: urn:request:7a32c34e-4525-4713-aed6-3436d9b48856 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:59,235 [238] [DEBUG] [app] Starting request: urn:request:f6c752e1-e976-4b9a-885a-3a0bdfbe989a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:48:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:48:59,238 [239] [DEBUG] [app] Starting request: urn:request:f9ba5a2c-b057-45bf-a07c-97d0b188b274 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:59,238 [246] [DEBUG] [app] Starting request: urn:request:3228a65d-0745-4e0c-802f-c19b18c0b077 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:48:59,238 [239] [DEBUG] [app] Ending request: urn:request:f9ba5a2c-b057-45bf-a07c-97d0b188b274 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9ba5a2c-b057-45bf-a07c-97d0b188b274', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:48:59,239 [246] [DEBUG] [app] Ending request: urn:request:3228a65d-0745-4e0c-802f-c19b18c0b077 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3228a65d-0745-4e0c-802f-c19b18c0b077', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:48:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:48:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:48:59,241 [238] [DEBUG] [app] Starting request: urn:request:672fcdc9-db3f-4bbd-95f1-1e8ad90a5bc3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:48:59,242 [238] [DEBUG] [app] Ending request: urn:request:672fcdc9-db3f-4bbd-95f1-1e8ad90a5bc3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:672fcdc9-db3f-4bbd-95f1-1e8ad90a5bc3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:59,242 [236] [DEBUG] [app] Starting request: urn:request:d27b2466-6a38-4c81-8a9d-2cdd03a074a7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:48:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:59,242 [236] [DEBUG] [app] Ending request: urn:request:d27b2466-6a38-4c81-8a9d-2cdd03a074a7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d27b2466-6a38-4c81-8a9d-2cdd03a074a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:48:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:48:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:48:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:48:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:48:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:48:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:48:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:48:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:48:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:48:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:48:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:48:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:48:59,253 [238] [DEBUG] [app] Ending request: urn:request:f6c752e1-e976-4b9a-885a-3a0bdfbe989a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f6c752e1-e976-4b9a-885a-3a0bdfbe989a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:48:59,253 [235] [DEBUG] [app] Ending request: urn:request:7a32c34e-4525-4713-aed6-3436d9b48856 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a32c34e-4525-4713-aed6-3436d9b48856', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:48:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:48:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:48:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 03:48:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:48:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:48:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 03:48:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:48:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:04.936688+00:00 (in 4.998345 seconds)
exportactionlogsworker stdout | 2025-02-07 03:48:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:59 UTC)" (scheduled at 2025-02-07 03:48:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:48:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:48:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:49:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:49:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:49:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:23.420886+00:00 (in 20.001340 seconds)
repositorygcworker stdout | 2025-02-07 03:49:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:03 UTC)" (scheduled at 2025-02-07 03:49:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:49:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:49:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 3, 419834), True, datetime.datetime(2025, 2, 7, 3, 49, 3, 419834), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:49:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:49:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:49:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:49:03,521 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:49:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:49:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:33.919286+00:00 (in 29.999515 seconds)
buildlogsarchiver stdout | 2025-02-07 03:49:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:33 UTC)" (scheduled at 2025-02-07 03:49:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:49:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 49, 3, 920031), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:49:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:49:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:49:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:49:04,482 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:49:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:49:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:52.937648+00:00 (in 48.000493 seconds)
exportactionlogsworker stdout | 2025-02-07 03:49:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:04 UTC)" (scheduled at 2025-02-07 03:49:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:49:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:49:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 4, 937498), True, datetime.datetime(2025, 2, 7, 3, 49, 4, 937498), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:49:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:49:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:49:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:04 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:49:08,589 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:49:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:49:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:18.732147+00:00 (in 9.999574 seconds)
notificationworker stdout | 2025-02-07 03:49:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:18 UTC)" (scheduled at 2025-02-07 03:49:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:49:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:49:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 8, 732883), True, datetime.datetime(2025, 2, 7, 3, 49, 8, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:49:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:49:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:49:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:49:09,172 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:49:09,707 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:49:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:49:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:24.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 03:49:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:40 UTC)" (scheduled at 2025-02-07 03:49:10.424250+00:00)
gcworker stdout | 2025-02-07 03:49:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:49:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899850431, None, 1, 0])
gcworker stdout | 2025-02-07 03:49:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:49:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:49:10,877 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 03:49:11,810 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
autopruneworker stdout | 2025-02-07 03:49:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:49:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:41.831521+00:00 (in 29.999550 seconds)
autopruneworker stdout | 2025-02-07 03:49:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:41 UTC)" (scheduled at 2025-02-07 03:49:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:49:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896551836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:49:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:49:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:49:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:41 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 03:49:11,914 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:49:12,915 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:13,241 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:49:13,962 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:49:14,007 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:49:14,044 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:49:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:49:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:29.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 03:49:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:29 UTC)" (scheduled at 2025-02-07 03:49:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:49:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:49:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:49:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:49:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:49:14,235 [238] [DEBUG] [app] Starting request: urn:request:8a86c1af-a15b-4b8e-b65a-1af04e8e77cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:14,235 [237] [DEBUG] [app] Starting request: urn:request:bcda0437-0c85-422b-aef5-f98e576c4693 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:49:14,238 [239] [DEBUG] [app] Starting request: urn:request:080a0d4a-7b7c-47ef-90dc-f11f05880cc7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:14,238 [246] [DEBUG] [app] Starting request: urn:request:59c1ec03-91f7-4bd2-b7ed-9c0412e39ab2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:14,239 [239] [DEBUG] [app] Ending request: urn:request:080a0d4a-7b7c-47ef-90dc-f11f05880cc7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:080a0d4a-7b7c-47ef-90dc-f11f05880cc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:49:14,239 [246] [DEBUG] [app] Ending request: urn:request:59c1ec03-91f7-4bd2-b7ed-9c0412e39ab2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59c1ec03-91f7-4bd2-b7ed-9c0412e39ab2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:49:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:49:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:14,242 [237] [DEBUG] [app] Starting request: urn:request:078cf34f-a9cc-432a-ae6e-3f92701c5e49 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:49:14,242 [235] [DEBUG] [app] Starting request: urn:request:8ef247e2-bb5e-421b-8cf2-3b702b46ae33 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:49:14,242 [237] [DEBUG] [app] Ending request: urn:request:078cf34f-a9cc-432a-ae6e-3f92701c5e49 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:078cf34f-a9cc-432a-ae6e-3f92701c5e49', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:14,243 [235] [DEBUG] [app] Ending request: urn:request:8ef247e2-bb5e-421b-8cf2-3b702b46ae33 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ef247e2-bb5e-421b-8cf2-3b702b46ae33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:49:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:49:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:49:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:49:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:14,255 [237] [DEBUG] [app] Ending request: urn:request:bcda0437-0c85-422b-aef5-f98e576c4693 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bcda0437-0c85-422b-aef5-f98e576c4693', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:49:14,255 [238] [DEBUG] [app] Ending request: urn:request:8a86c1af-a15b-4b8e-b65a-1af04e8e77cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a86c1af-a15b-4b8e-b65a-1af04e8e77cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:49:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:49:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
repositorygcworker stdout | 2025-02-07 03:49:14,450 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:49:14,918 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:49:14,990 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:49:15,047 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:49:16,090 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:49:16,093 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:49:16,096 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:49:16,099 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:49:16,102 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:49:16,669 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:49:16,945 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:49:17,117 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:49:17,122 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:49:17,125 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:49:17,199 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:49:18,123 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:49:18,131 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:49:18,133 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:49:18,136 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:49:18,138 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:49:18,142 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:49:18,145 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:49:18,147 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:49:18,150 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:49:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:49:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:28.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 03:49:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:28 UTC)" (scheduled at 2025-02-07 03:49:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:49:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:49:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 18, 732918), True, datetime.datetime(2025, 2, 7, 3, 49, 18, 732918), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:49:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:49:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:49:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:49:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:49:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:03.419110+00:00 (in 39.997755 seconds)
repositorygcworker stdout | 2025-02-07 03:49:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:23 UTC)" (scheduled at 2025-02-07 03:49:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:49:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:49:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:49:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:49:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:40.424250+00:00 (in 16.000746 seconds)
gcworker stdout | 2025-02-07 03:49:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:54 UTC)" (scheduled at 2025-02-07 03:49:24.423055+00:00)
gcworker stdout | 2025-02-07 03:49:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:49:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:49:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:49:26,234 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:49:28,199 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:49:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:49:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:38.732147+00:00 (in 9.999477 seconds)
notificationworker stdout | 2025-02-07 03:49:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:38 UTC)" (scheduled at 2025-02-07 03:49:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:49:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:49:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 28, 732985), True, datetime.datetime(2025, 2, 7, 3, 49, 28, 732985), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:49:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:49:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:49:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:49:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:49:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:44.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 03:49:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:44 UTC)" (scheduled at 2025-02-07 03:49:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:49:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:49:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:49:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:49:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:44 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 03:49:29,234 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
gunicorn-web stdout | 2025-02-07 03:49:29,235 [237] [DEBUG] [app] Starting request: urn:request:eb10633c-8b5e-490e-8778-0e2bb34331ec (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:29,235 [238] [DEBUG] [app] Starting request: urn:request:20a16299-3c09-4d2f-bb03-6b0cb221fc31 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:49:29,238 [246] [DEBUG] [app] Starting request: urn:request:debe5102-cdd7-4f3e-a5fc-9ce4a242e24e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:29,239 [245] [DEBUG] [app] Starting request: urn:request:8b1624d3-d170-4bfe-9304-e5ffb9afd42d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:29,239 [246] [DEBUG] [app] Ending request: urn:request:debe5102-cdd7-4f3e-a5fc-9ce4a242e24e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:debe5102-cdd7-4f3e-a5fc-9ce4a242e24e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:49:29,239 [245] [DEBUG] [app] Ending request: urn:request:8b1624d3-d170-4bfe-9304-e5ffb9afd42d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b1624d3-d170-4bfe-9304-e5ffb9afd42d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:49:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:49:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:49:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:29,242 [238] [DEBUG] [app] Starting request: urn:request:390cbdc0-f8fe-4ffd-a986-c7e56abe5ba8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:49:29,242 [238] [DEBUG] [app] Ending request: urn:request:390cbdc0-f8fe-4ffd-a986-c7e56abe5ba8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:390cbdc0-f8fe-4ffd-a986-c7e56abe5ba8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:29,243 [235] [DEBUG] [app] Starting request: urn:request:a9d55d6b-c310-4712-aa28-3bbb5debb887 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:49:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:29,243 [235] [DEBUG] [app] Ending request: urn:request:a9d55d6b-c310-4712-aa28-3bbb5debb887 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9d55d6b-c310-4712-aa28-3bbb5debb887', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:49:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:49:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:29,254 [238] [DEBUG] [app] Ending request: urn:request:20a16299-3c09-4d2f-bb03-6b0cb221fc31 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20a16299-3c09-4d2f-bb03-6b0cb221fc31', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:49:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:49:29,255 [237] [DEBUG] [app] Ending request: urn:request:eb10633c-8b5e-490e-8778-0e2bb34331ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb10633c-8b5e-490e-8778-0e2bb34331ec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:49:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 03:49:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:49:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:32.625945+00:00 (in 1.004279 seconds)
namespacegcworker stdout | 2025-02-07 03:49:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:31 UTC)" (scheduled at 2025-02-07 03:49:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:49:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:49:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 31, 621978), True, datetime.datetime(2025, 2, 7, 3, 49, 31, 621978), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:49:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:49:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:49:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:49:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:49:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:31.621131+00:00 (in 58.994615 seconds)
namespacegcworker stdout | 2025-02-07 03:49:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:32 UTC)" (scheduled at 2025-02-07 03:49:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:49:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:49:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:49:33,558 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:49:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:49:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:03.919286+00:00 (in 29.999511 seconds)
buildlogsarchiver stdout | 2025-02-07 03:49:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:03 UTC)" (scheduled at 2025-02-07 03:49:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:49:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 49, 33, 920041), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:49:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:49:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:49:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:49:34,512 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:56.713547+00:00 (in 18.997155 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:37 UTC)" (scheduled at 2025-02-07 03:49:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:37 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:49:38,622 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:49:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:49:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:48.732147+00:00 (in 9.999491 seconds)
notificationworker stdout | 2025-02-07 03:49:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:48 UTC)" (scheduled at 2025-02-07 03:49:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:49:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:49:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 38, 732958), True, datetime.datetime(2025, 2, 7, 3, 49, 38, 732958), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:49:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:49:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:49:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:49:39,209 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:49:39,744 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:49:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:49:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:54.423055+00:00 (in 13.998391 seconds)
gcworker stdout | 2025-02-07 03:49:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:10 UTC)" (scheduled at 2025-02-07 03:49:40.424250+00:00)
gcworker stdout | 2025-02-07 03:49:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:49:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899880431, None, 1, 0])
gcworker stdout | 2025-02-07 03:49:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:49:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:49:40,910 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:49:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:49:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:11.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 03:49:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:11 UTC)" (scheduled at 2025-02-07 03:49:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:49:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896581836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:49:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:49:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:49:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:49:41,846 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:49:41,936 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:49:42,937 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:43,277 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:49:43,999 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:49:44,031 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:49:44,079 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:49:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:49:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:59.225378+00:00 (in 14.999425 seconds)
quotatotalworker stdout | 2025-02-07 03:49:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:59 UTC)" (scheduled at 2025-02-07 03:49:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:49:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:49:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:49:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:49:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:49:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:49:44,235 [238] [DEBUG] [app] Starting request: urn:request:1d88784f-73ec-441b-bff2-b7aecacc9ed8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:44,235 [235] [DEBUG] [app] Starting request: urn:request:00220914-1f8a-48de-ae12-b3735e1f6bb8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:49:44,238 [246] [DEBUG] [app] Starting request: urn:request:e90864fa-c5c0-48c8-8a86-daa8ce5ff9ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:44,238 [243] [DEBUG] [app] Starting request: urn:request:2d16826c-dc0e-47ff-ad40-df962fbaad1a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:44,239 [243] [DEBUG] [app] Ending request: urn:request:2d16826c-dc0e-47ff-ad40-df962fbaad1a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d16826c-dc0e-47ff-ad40-df962fbaad1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:49:44,239 [246] [DEBUG] [app] Ending request: urn:request:e90864fa-c5c0-48c8-8a86-daa8ce5ff9ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e90864fa-c5c0-48c8-8a86-daa8ce5ff9ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:49:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 03:49:44,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 162 0.003)
gunicorn-web stdout | 2025-02-07 03:49:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:44,243 [238] [DEBUG] [app] Starting request: urn:request:49aad377-9589-4afb-b49d-de40d22e165b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:49:44,244 [238] [DEBUG] [app] Ending request: urn:request:49aad377-9589-4afb-b49d-de40d22e165b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:49aad377-9589-4afb-b49d-de40d22e165b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:44,244 [237] [DEBUG] [app] Starting request: urn:request:db2e1b24-0752-42eb-b13a-9643f8ffcdf1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:49:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:49:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:44,245 [237] [DEBUG] [app] Ending request: urn:request:db2e1b24-0752-42eb-b13a-9643f8ffcdf1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db2e1b24-0752-42eb-b13a-9643f8ffcdf1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:49:44,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:44,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:44,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:44,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:49:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:44,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:44,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:44,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:44,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:44,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:44,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:44,257 [238] [DEBUG] [app] Ending request: urn:request:1d88784f-73ec-441b-bff2-b7aecacc9ed8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1d88784f-73ec-441b-bff2-b7aecacc9ed8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:49:44,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:44,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:49:44,257 [235] [DEBUG] [app] Ending request: urn:request:00220914-1f8a-48de-ae12-b3735e1f6bb8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:00220914-1f8a-48de-ae12-b3735e1f6bb8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.022)
gunicorn-web stdout | 2025-02-07 03:49:44,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:44,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.023)
repositorygcworker stdout | 2025-02-07 03:49:44,486 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:49:44,932 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:49:45,027 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:49:45,078 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:49:46,098 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:49:46,100 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:49:46,103 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:49:46,107 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:49:46,110 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:49:46,705 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:49:46,978 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:49:47,127 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:49:47,130 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:49:47,133 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:49:47,235 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:49:48,131 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:49:48,138 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:49:48,143 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:49:48,146 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:49:48,148 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:49:48,151 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:49:48,155 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:49:48,157 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:49:48,160 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:49:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:49:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:50.814677+00:00 (in 2.082026 seconds)
notificationworker stdout | 2025-02-07 03:49:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:58 UTC)" (scheduled at 2025-02-07 03:49:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:49:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:49:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 48, 732934), True, datetime.datetime(2025, 2, 7, 3, 49, 48, 732934), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:49:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:49:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:49:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:49:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:49:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:49:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:58.732147+00:00 (in 7.916993 seconds)
notificationworker stdout | 2025-02-07 03:49:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:50 UTC)" (scheduled at 2025-02-07 03:49:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:49:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:49:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:49:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:49:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:49:59.937899+00:00 (in 6.999782 seconds)
exportactionlogsworker stdout | 2025-02-07 03:49:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:54:52 UTC)" (scheduled at 2025-02-07 03:49:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:49:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 49, 52, 938382), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:49:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 49, 52, 938382), True, datetime.datetime(2025, 2, 7, 3, 49, 52, 938382), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:49:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 49, 52, 938382), True, datetime.datetime(2025, 2, 7, 3, 49, 52, 938382), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 49, 52, 938382), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:49:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:49:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:54:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:49:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:49:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:10.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 03:49:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:24 UTC)" (scheduled at 2025-02-07 03:49:54.423055+00:00)
gcworker stdout | 2025-02-07 03:49:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:49:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:49:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:49:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:56.118855+00:00 (in 59.999465 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:49:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:56 UTC)" (scheduled at 2025-02-07 03:49:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:49:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:49:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:49:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:49:56,261 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:37.715878+00:00 (in 41.001846 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:56 UTC)" (scheduled at 2025-02-07 03:49:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:49:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:49:58,225 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:49:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:49:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:08.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 03:49:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:08 UTC)" (scheduled at 2025-02-07 03:49:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:49:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:49:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 49, 58, 732885), True, datetime.datetime(2025, 2, 7, 3, 49, 58, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:49:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:49:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:49:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:49:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:49:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:14.225378+00:00 (in 14.999491 seconds)
quotatotalworker stdout | 2025-02-07 03:49:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:14 UTC)" (scheduled at 2025-02-07 03:49:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:49:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:49:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:49:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:49:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:49:59,235 [237] [DEBUG] [app] Starting request: urn:request:ab611386-1f21-4f1e-a350-8be3930ec0ae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:59,235 [235] [DEBUG] [app] Starting request: urn:request:e4bde83f-a2ed-4db1-9456-a52fa449ce67 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:49:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:49:59,238 [245] [DEBUG] [app] Starting request: urn:request:0cee115e-38c8-44fe-acad-49db204c96ee (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:59,238 [239] [DEBUG] [app] Starting request: urn:request:1aba47ec-4930-47c3-b233-29c0dbe6eb12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:49:59,239 [239] [DEBUG] [app] Ending request: urn:request:1aba47ec-4930-47c3-b233-29c0dbe6eb12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1aba47ec-4930-47c3-b233-29c0dbe6eb12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:49:59,239 [245] [DEBUG] [app] Ending request: urn:request:0cee115e-38c8-44fe-acad-49db204c96ee (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0cee115e-38c8-44fe-acad-49db204c96ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:49:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:49:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:49:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:49:59,242 [237] [DEBUG] [app] Starting request: urn:request:22fff3bd-d057-459e-9d3b-5f2af797335a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:49:59,242 [237] [DEBUG] [app] Ending request: urn:request:22fff3bd-d057-459e-9d3b-5f2af797335a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:22fff3bd-d057-459e-9d3b-5f2af797335a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:59,242 [235] [DEBUG] [app] Starting request: urn:request:3d3bf0a3-e9f9-44d3-a6e5-bba7fc529894 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:49:59,242 [235] [DEBUG] [app] Ending request: urn:request:3d3bf0a3-e9f9-44d3-a6e5-bba7fc529894 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d3bf0a3-e9f9-44d3-a6e5-bba7fc529894', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:49:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:49:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:49:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:49:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:49:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:49:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:49:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:49:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:49:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:49:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:49:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:49:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:49:59,254 [235] [DEBUG] [app] Ending request: urn:request:e4bde83f-a2ed-4db1-9456-a52fa449ce67 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4bde83f-a2ed-4db1-9456-a52fa449ce67', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:49:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:59,255 [237] [DEBUG] [app] Ending request: urn:request:ab611386-1f21-4f1e-a350-8be3930ec0ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab611386-1f21-4f1e-a350-8be3930ec0ae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:49:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:49:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:49:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:49:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:49:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
queuecleanupworker stdout | 2025-02-07 03:49:59,270 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:49:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:49:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:04.936688+00:00 (in 4.998294 seconds)
exportactionlogsworker stdout | 2025-02-07 03:49:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:59 UTC)" (scheduled at 2025-02-07 03:49:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:49:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:49:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:50:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:50:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:50:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:23.420886+00:00 (in 20.001322 seconds)
repositorygcworker stdout | 2025-02-07 03:50:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:03 UTC)" (scheduled at 2025-02-07 03:50:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:50:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:50:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 3, 419874), True, datetime.datetime(2025, 2, 7, 3, 50, 3, 419874), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:50:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:50:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:50:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:50:03,579 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:50:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:50:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:33.919286+00:00 (in 29.999488 seconds)
buildlogsarchiver stdout | 2025-02-07 03:50:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:33 UTC)" (scheduled at 2025-02-07 03:50:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:50:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 50, 3, 920083), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:50:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:50:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:50:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:50:04,520 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:50:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:50:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:59.937899+00:00 (in 55.000751 seconds)
exportactionlogsworker stdout | 2025-02-07 03:50:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:04 UTC)" (scheduled at 2025-02-07 03:50:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:50:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:50:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 4, 937457), True, datetime.datetime(2025, 2, 7, 3, 50, 4, 937457), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:50:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:50:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:50:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:04 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:50:08,641 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:50:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:50:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:18.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 03:50:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:18 UTC)" (scheduled at 2025-02-07 03:50:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:50:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:50:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 8, 732868), True, datetime.datetime(2025, 2, 7, 3, 50, 8, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:50:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:50:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:50:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:50:09,245 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:50:09,780 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:50:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:50:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:24.423055+00:00 (in 13.998340 seconds)
gcworker stdout | 2025-02-07 03:50:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:40 UTC)" (scheduled at 2025-02-07 03:50:10.424250+00:00)
gcworker stdout | 2025-02-07 03:50:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:50:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899910431, None, 1, 0])
gcworker stdout | 2025-02-07 03:50:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:50:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:50:10,935 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:50:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:50:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:41.831521+00:00 (in 29.999517 seconds)
autopruneworker stdout | 2025-02-07 03:50:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:41 UTC)" (scheduled at 2025-02-07 03:50:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:50:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896611836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:50:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:50:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:50:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:50:11,872 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:50:11,973 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:50:12,973 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:13,313 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:50:14,027 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:50:14,056 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:50:14,089 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:50:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:50:14,226 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:29.225378+00:00 (in 14.999353 seconds)
quotatotalworker stdout | 2025-02-07 03:50:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:29 UTC)" (scheduled at 2025-02-07 03:50:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:50:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:50:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:50:14,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:50:14,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:50:14,236 [238] [DEBUG] [app] Starting request: urn:request:a83b2c3c-b50d-4b3f-a508-0578bf681c30 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:14,236 [235] [DEBUG] [app] Starting request: urn:request:6d008737-2231-493f-a821-791a9a843a6d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:50:14,239 [246] [DEBUG] [app] Starting request: urn:request:9f25f8c7-b88e-4ba1-9374-87b06520132a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:14,240 [239] [DEBUG] [app] Starting request: urn:request:bf5fb516-1af4-4ef3-9f90-a60945e8e4b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:14,240 [246] [DEBUG] [app] Ending request: urn:request:9f25f8c7-b88e-4ba1-9374-87b06520132a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9f25f8c7-b88e-4ba1-9374-87b06520132a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:14,240 [239] [DEBUG] [app] Ending request: urn:request:bf5fb516-1af4-4ef3-9f90-a60945e8e4b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf5fb516-1af4-4ef3-9f90-a60945e8e4b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:14,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:50:14,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:14,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:14,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:50:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:14,244 [235] [DEBUG] [app] Starting request: urn:request:223d2f37-5022-476e-bc51-472b358c7a9b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:50:14,244 [235] [DEBUG] [app] Ending request: urn:request:223d2f37-5022-476e-bc51-472b358c7a9b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:223d2f37-5022-476e-bc51-472b358c7a9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:50:14,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:14,244 [236] [DEBUG] [app] Starting request: urn:request:86e2dd18-2424-47c4-8ada-af8e6f6c6c7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:50:14,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:14,245 [236] [DEBUG] [app] Ending request: urn:request:86e2dd18-2424-47c4-8ada-af8e6f6c6c7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:86e2dd18-2424-47c4-8ada-af8e6f6c6c7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:50:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:50:14,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:14,246 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:50:14,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:14,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:50:14,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:14,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:14,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:14,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:14,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:14,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:14,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:14,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:14,257 [235] [DEBUG] [app] Ending request: urn:request:6d008737-2231-493f-a821-791a9a843a6d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d008737-2231-493f-a821-791a9a843a6d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:14,257 [238] [DEBUG] [app] Ending request: urn:request:a83b2c3c-b50d-4b3f-a508-0578bf681c30 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a83b2c3c-b50d-4b3f-a508-0578bf681c30', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:14,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:50:14,257 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.024)
gunicorn-web stdout | 2025-02-07 03:50:14,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:50:14,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
repositorygcworker stdout | 2025-02-07 03:50:14,513 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:50:14,950 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:50:15,063 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:50:15,088 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:50:16,105 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:50:16,109 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:50:16,113 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:50:16,115 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:50:16,118 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:50:16,734 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:50:17,015 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:50:17,136 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:50:17,139 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:50:17,141 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:50:17,273 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:50:18,143 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:50:18,148 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:50:18,154 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:50:18,157 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:50:18,159 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:50:18,163 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:50:18,166 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:50:18,168 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:50:18,171 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:50:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:50:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 03:50:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:28 UTC)" (scheduled at 2025-02-07 03:50:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:50:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:50:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 18, 732870), True, datetime.datetime(2025, 2, 7, 3, 50, 18, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:50:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:50:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:50:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:50:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:50:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:03.419110+00:00 (in 39.997749 seconds)
repositorygcworker stdout | 2025-02-07 03:50:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:23 UTC)" (scheduled at 2025-02-07 03:50:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:50:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:50:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:50:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:50:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:40.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 03:50:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:54 UTC)" (scheduled at 2025-02-07 03:50:24.423055+00:00)
gcworker stdout | 2025-02-07 03:50:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:50:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:50:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:50:26,275 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:50:28,234 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:50:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:50:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:38.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 03:50:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:38 UTC)" (scheduled at 2025-02-07 03:50:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:50:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:50:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 28, 732864), True, datetime.datetime(2025, 2, 7, 3, 50, 28, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:50:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:50:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:50:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:50:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:50:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:44.225378+00:00 (in 14.999477 seconds)
quotatotalworker stdout | 2025-02-07 03:50:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:44 UTC)" (scheduled at 2025-02-07 03:50:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:50:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:50:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:50:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:50:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:50:29,235 [238] [DEBUG] [app] Starting request: urn:request:61304844-e366-4489-b044-fef7733e9970 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:29,235 [235] [DEBUG] [app] Starting request: urn:request:24c496cc-2d8e-483d-81cc-67e2a33ae597 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:50:29,238 [246] [DEBUG] [app] Starting request: urn:request:10c2db1e-1d5a-4745-8677-8022d03cbf98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:29,238 [239] [DEBUG] [app] Starting request: urn:request:07354e0e-759c-484f-a6ef-d6b2c859b2c4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:29,238 [239] [DEBUG] [app] Ending request: urn:request:07354e0e-759c-484f-a6ef-d6b2c859b2c4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07354e0e-759c-484f-a6ef-d6b2c859b2c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:29,238 [246] [DEBUG] [app] Ending request: urn:request:10c2db1e-1d5a-4745-8677-8022d03cbf98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:10c2db1e-1d5a-4745-8677-8022d03cbf98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:50:29,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:50:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:29,241 [238] [DEBUG] [app] Starting request: urn:request:f86f4139-826e-4754-8c4a-18dc4bd409e5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:50:29,241 [238] [DEBUG] [app] Ending request: urn:request:f86f4139-826e-4754-8c4a-18dc4bd409e5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f86f4139-826e-4754-8c4a-18dc4bd409e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:50:29,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:29,241 [237] [DEBUG] [app] Starting request: urn:request:7dd1d46d-3b1a-4dcc-a0c4-d37d0240c9d8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:50:29,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:29,242 [237] [DEBUG] [app] Ending request: urn:request:7dd1d46d-3b1a-4dcc-a0c4-d37d0240c9d8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7dd1d46d-3b1a-4dcc-a0c4-d37d0240c9d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:50:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:50:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:50:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:29,252 [235] [DEBUG] [app] Ending request: urn:request:24c496cc-2d8e-483d-81cc-67e2a33ae597 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:24c496cc-2d8e-483d-81cc-67e2a33ae597', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:50:29,253 [238] [DEBUG] [app] Ending request: urn:request:61304844-e366-4489-b044-fef7733e9970 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61304844-e366-4489-b044-fef7733e9970', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:50:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:50:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 03:50:29,307 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:50:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:50:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:32.625945+00:00 (in 1.004335 seconds)
namespacegcworker stdout | 2025-02-07 03:50:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:31 UTC)" (scheduled at 2025-02-07 03:50:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:50:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:50:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 31, 621885), True, datetime.datetime(2025, 2, 7, 3, 50, 31, 621885), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:50:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:50:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:50:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:50:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:50:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:31.621131+00:00 (in 58.994712 seconds)
namespacegcworker stdout | 2025-02-07 03:50:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:32 UTC)" (scheduled at 2025-02-07 03:50:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:50:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:50:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:50:33,615 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:50:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:50:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:03.919286+00:00 (in 29.999476 seconds)
buildlogsarchiver stdout | 2025-02-07 03:50:33,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:03 UTC)" (scheduled at 2025-02-07 03:50:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:50:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 50, 33, 920104), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:50:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:50:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:50:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:50:34,556 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:56.713547+00:00 (in 18.997181 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:37 UTC)" (scheduled at 2025-02-07 03:50:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:37 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:50:38,669 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:50:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:50:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:48.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 03:50:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:48 UTC)" (scheduled at 2025-02-07 03:50:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:50:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:50:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 38, 732842), True, datetime.datetime(2025, 2, 7, 3, 50, 38, 732842), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:50:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:50:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:50:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:50:39,279 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:50:39,794 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:50:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:50:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:54.423055+00:00 (in 13.998356 seconds)
gcworker stdout | 2025-02-07 03:50:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:10 UTC)" (scheduled at 2025-02-07 03:50:40.424250+00:00)
gcworker stdout | 2025-02-07 03:50:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:50:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899940430, None, 1, 0])
gcworker stdout | 2025-02-07 03:50:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:50:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:50:40,971 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:50:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:50:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:11.831521+00:00 (in 29.999544 seconds)
autopruneworker stdout | 2025-02-07 03:50:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:11 UTC)" (scheduled at 2025-02-07 03:50:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:50:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896641836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:50:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:50:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:50:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:50:41,883 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:50:42,009 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:50:43,010 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:43,349 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:50:44,037 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:50:44,089 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:50:44,125 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:50:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:50:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:59.225378+00:00 (in 14.999543 seconds)
quotatotalworker stdout | 2025-02-07 03:50:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:59 UTC)" (scheduled at 2025-02-07 03:50:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:50:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:50:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:50:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:50:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:50:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:50:44,235 [237] [DEBUG] [app] Starting request: urn:request:c6671ade-341c-4abc-b0f0-1e279fb1f664 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:44,235 [235] [DEBUG] [app] Starting request: urn:request:a150660d-bcda-48ed-aa13-fa01978be5c4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:50:44,239 [239] [DEBUG] [app] Starting request: urn:request:501c9c22-4c70-4d16-95be-50ee9d0dde0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:44,239 [246] [DEBUG] [app] Starting request: urn:request:57d07f95-cdd5-4c86-a6e0-59b2a0077dbd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:44,239 [239] [DEBUG] [app] Ending request: urn:request:501c9c22-4c70-4d16-95be-50ee9d0dde0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:501c9c22-4c70-4d16-95be-50ee9d0dde0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:44,240 [246] [DEBUG] [app] Ending request: urn:request:57d07f95-cdd5-4c86-a6e0-59b2a0077dbd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57d07f95-cdd5-4c86-a6e0-59b2a0077dbd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:50:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:50:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:44,243 [238] [DEBUG] [app] Starting request: urn:request:d8c244c7-ed80-4463-917e-59c1d59480ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:50:44,243 [236] [DEBUG] [app] Starting request: urn:request:73873897-6005-4a25-8a5c-53a667125c93 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:50:44,244 [236] [DEBUG] [app] Ending request: urn:request:73873897-6005-4a25-8a5c-53a667125c93 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73873897-6005-4a25-8a5c-53a667125c93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:50:44,244 [238] [DEBUG] [app] Ending request: urn:request:d8c244c7-ed80-4463-917e-59c1d59480ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8c244c7-ed80-4463-917e-59c1d59480ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:50:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:50:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:50:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:50:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:44,255 [235] [DEBUG] [app] Ending request: urn:request:a150660d-bcda-48ed-aa13-fa01978be5c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a150660d-bcda-48ed-aa13-fa01978be5c4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:44,256 [237] [DEBUG] [app] Ending request: urn:request:c6671ade-341c-4abc-b0f0-1e279fb1f664 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c6671ade-341c-4abc-b0f0-1e279fb1f664', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:50:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:50:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:50:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
repositorygcworker stdout | 2025-02-07 03:50:44,549 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:50:44,960 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:50:45,098 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:50:45,101 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:50:46,113 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:50:46,116 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:50:46,118 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:50:46,122 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:50:46,125 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:50:46,771 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:50:47,048 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:50:47,144 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:50:47,147 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:50:47,150 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:50:47,290 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:50:48,156 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:50:48,159 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:50:48,165 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:50:48,168 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:50:48,171 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:50:48,175 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:50:48,178 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:50:48,181 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:50:48,183 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:50:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:50:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:50.814677+00:00 (in 2.082090 seconds)
notificationworker stdout | 2025-02-07 03:50:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:58 UTC)" (scheduled at 2025-02-07 03:50:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:50:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:50:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 48, 732857), True, datetime.datetime(2025, 2, 7, 3, 50, 48, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:50:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:50:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:50:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:50:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:50:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:50:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:50:58.732147+00:00 (in 7.917015 seconds)
notificationworker stdout | 2025-02-07 03:50:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:50 UTC)" (scheduled at 2025-02-07 03:50:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:50:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:50:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:50:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:50:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:10.424250+00:00 (in 16.000715 seconds)
gcworker stdout | 2025-02-07 03:50:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:24 UTC)" (scheduled at 2025-02-07 03:50:54.423055+00:00)
gcworker stdout | 2025-02-07 03:50:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:50:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:50:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:50:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:56.118855+00:00 (in 59.999489 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:50:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:56 UTC)" (scheduled at 2025-02-07 03:50:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:50:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:50:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:50:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:50:56,284 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:37.715878+00:00 (in 41.001840 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:56 UTC)" (scheduled at 2025-02-07 03:50:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:50:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:50:58,248 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:50:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:50:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:08.732147+00:00 (in 9.999540 seconds)
notificationworker stdout | 2025-02-07 03:50:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:08 UTC)" (scheduled at 2025-02-07 03:50:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:50:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:50:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 50, 58, 732885), True, datetime.datetime(2025, 2, 7, 3, 50, 58, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:50:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:50:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:50:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:50:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:50:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:14.225378+00:00 (in 14.999519 seconds)
quotatotalworker stdout | 2025-02-07 03:50:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:14 UTC)" (scheduled at 2025-02-07 03:50:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:50:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:50:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:50:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:50:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:50:59,236 [236] [DEBUG] [app] Starting request: urn:request:4b644b6e-3cb9-43e1-b339-d1179a6a8ae9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:59,236 [235] [DEBUG] [app] Starting request: urn:request:415a6ded-4333-49b7-a8f9-edee5bf5c2c4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:50:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:50:59,239 [239] [DEBUG] [app] Starting request: urn:request:708f59ef-6d70-4ffc-8eb0-5cf4fef62da0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:59,239 [246] [DEBUG] [app] Starting request: urn:request:7627a0d8-15f8-42d5-9807-2f5a481a5ac3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:50:59,239 [239] [DEBUG] [app] Ending request: urn:request:708f59ef-6d70-4ffc-8eb0-5cf4fef62da0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:708f59ef-6d70-4ffc-8eb0-5cf4fef62da0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:59,239 [246] [DEBUG] [app] Ending request: urn:request:7627a0d8-15f8-42d5-9807-2f5a481a5ac3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7627a0d8-15f8-42d5-9807-2f5a481a5ac3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:50:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:50:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:50:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:50:59,242 [236] [DEBUG] [app] Starting request: urn:request:fa61c9d8-c23a-45aa-819e-201012d75856 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:50:59,242 [237] [DEBUG] [app] Starting request: urn:request:351cb821-afad-443e-979b-5ca603f77d5e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:50:59,243 [236] [DEBUG] [app] Ending request: urn:request:fa61c9d8-c23a-45aa-819e-201012d75856 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa61c9d8-c23a-45aa-819e-201012d75856', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:50:59,243 [237] [DEBUG] [app] Ending request: urn:request:351cb821-afad-443e-979b-5ca603f77d5e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:351cb821-afad-443e-979b-5ca603f77d5e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:50:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:50:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:50:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:50:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:50:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:50:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:50:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:50:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:50:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:50:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:50:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:50:59,255 [235] [DEBUG] [app] Ending request: urn:request:415a6ded-4333-49b7-a8f9-edee5bf5c2c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:415a6ded-4333-49b7-a8f9-edee5bf5c2c4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:50:59,255 [236] [DEBUG] [app] Ending request: urn:request:4b644b6e-3cb9-43e1-b339-d1179a6a8ae9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4b644b6e-3cb9-43e1-b339-d1179a6a8ae9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:50:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:50:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:50:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:50:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:50:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 03:50:59,343 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:50:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:50:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:04.936688+00:00 (in 4.998278 seconds)
exportactionlogsworker stdout | 2025-02-07 03:50:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:59 UTC)" (scheduled at 2025-02-07 03:50:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:50:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:50:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:51:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:51:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:51:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:23.420886+00:00 (in 20.001318 seconds)
repositorygcworker stdout | 2025-02-07 03:51:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:03 UTC)" (scheduled at 2025-02-07 03:51:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:51:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:51:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 3, 419846), True, datetime.datetime(2025, 2, 7, 3, 51, 3, 419846), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:51:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:51:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:51:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:51:03,645 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:51:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:51:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:33.919286+00:00 (in 29.999476 seconds)
buildlogsarchiver stdout | 2025-02-07 03:51:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:33 UTC)" (scheduled at 2025-02-07 03:51:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:51:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 51, 3, 920083), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:51:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:51:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:51:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:51:04,570 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:51:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:51:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:59.937899+00:00 (in 55.000757 seconds)
exportactionlogsworker stdout | 2025-02-07 03:51:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:04 UTC)" (scheduled at 2025-02-07 03:51:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:51:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:51:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 4, 937451), True, datetime.datetime(2025, 2, 7, 3, 51, 4, 937451), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:51:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:51:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:51:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:04 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:51:08,690 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:51:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:51:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:18.732147+00:00 (in 9.999435 seconds)
notificationworker stdout | 2025-02-07 03:51:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:18 UTC)" (scheduled at 2025-02-07 03:51:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:51:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:51:08,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 8, 733110), True, datetime.datetime(2025, 2, 7, 3, 51, 8, 733110), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:51:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:51:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:51:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:51:09,292 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:51:09,832 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:51:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:51:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:24.423055+00:00 (in 13.998344 seconds)
gcworker stdout | 2025-02-07 03:51:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:40 UTC)" (scheduled at 2025-02-07 03:51:10.424250+00:00)
gcworker stdout | 2025-02-07 03:51:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:51:10,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738899970431, None, 1, 0])
gcworker stdout | 2025-02-07 03:51:10,434 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:51:10,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:51:11,007 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:51:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:51:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 03:51:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:41 UTC)" (scheduled at 2025-02-07 03:51:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:51:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896671836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:51:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:51:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:51:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:51:11,901 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:51:12,042 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:51:13,046 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:13,386 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:51:14,073 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:51:14,114 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:51:14,137 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:51:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:51:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:29.225378+00:00 (in 14.999514 seconds)
quotatotalworker stdout | 2025-02-07 03:51:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:29 UTC)" (scheduled at 2025-02-07 03:51:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:51:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:51:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:51:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:51:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:51:14,236 [235] [DEBUG] [app] Starting request: urn:request:875a0537-8435-4a35-a788-a08f6ddc3cc6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:14,236 [237] [DEBUG] [app] Starting request: urn:request:1011f4b4-1ad9-400b-ae5b-52fc01061116 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:51:14,239 [239] [DEBUG] [app] Starting request: urn:request:d95ddc89-5bd1-47fd-a4d2-699f6af8317e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:14,239 [244] [DEBUG] [app] Starting request: urn:request:1c52bd6e-e2bb-4a7b-ac13-e8b58cdcf129 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:14,239 [239] [DEBUG] [app] Ending request: urn:request:d95ddc89-5bd1-47fd-a4d2-699f6af8317e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d95ddc89-5bd1-47fd-a4d2-699f6af8317e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:51:14,239 [244] [DEBUG] [app] Ending request: urn:request:1c52bd6e-e2bb-4a7b-ac13-e8b58cdcf129 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c52bd6e-e2bb-4a7b-ac13-e8b58cdcf129', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:51:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:51:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:51:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:14,243 [238] [DEBUG] [app] Starting request: urn:request:9eb20cb3-3e35-4c2c-8bee-f0113ef9a702 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:14,243 [236] [DEBUG] [app] Starting request: urn:request:1fc4f842-d8a1-4fb3-974c-f86d934d7f81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:14,243 [236] [DEBUG] [app] Ending request: urn:request:1fc4f842-d8a1-4fb3-974c-f86d934d7f81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1fc4f842-d8a1-4fb3-974c-f86d934d7f81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:51:14,243 [238] [DEBUG] [app] Ending request: urn:request:9eb20cb3-3e35-4c2c-8bee-f0113ef9a702 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9eb20cb3-3e35-4c2c-8bee-f0113ef9a702', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:51:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:51:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:14,254 [235] [DEBUG] [app] Ending request: urn:request:875a0537-8435-4a35-a788-a08f6ddc3cc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:875a0537-8435-4a35-a788-a08f6ddc3cc6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:51:14,254 [237] [DEBUG] [app] Ending request: urn:request:1011f4b4-1ad9-400b-ae5b-52fc01061116 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1011f4b4-1ad9-400b-ae5b-52fc01061116', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:51:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:51:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositorygcworker stdout | 2025-02-07 03:51:14,582 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:51:14,996 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:51:15,131 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:51:15,136 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:51:16,123 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:51:16,128 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:51:16,132 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:51:16,136 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:51:16,139 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:51:16,790 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:51:17,084 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:51:17,152 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:51:17,155 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:51:17,158 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:51:17,326 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:51:18,168 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:51:18,171 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:51:18,175 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:51:18,178 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:51:18,181 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:51:18,185 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:51:18,188 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:51:18,191 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:51:18,194 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:51:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:51:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:28.732147+00:00 (in 9.999478 seconds)
notificationworker stdout | 2025-02-07 03:51:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:28 UTC)" (scheduled at 2025-02-07 03:51:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:51:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:51:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 18, 732989), True, datetime.datetime(2025, 2, 7, 3, 51, 18, 732989), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:51:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:51:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:51:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:51:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:51:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:03.419110+00:00 (in 39.997741 seconds)
repositorygcworker stdout | 2025-02-07 03:51:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:23 UTC)" (scheduled at 2025-02-07 03:51:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:51:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:51:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:51:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:51:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:40.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 03:51:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:54 UTC)" (scheduled at 2025-02-07 03:51:24.423055+00:00)
gcworker stdout | 2025-02-07 03:51:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:51:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:51:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:51:26,320 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:51:28,274 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:51:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:51:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:38.732147+00:00 (in 9.999486 seconds)
notificationworker stdout | 2025-02-07 03:51:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:38 UTC)" (scheduled at 2025-02-07 03:51:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:51:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:51:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 28, 732965), True, datetime.datetime(2025, 2, 7, 3, 51, 28, 732965), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:51:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:51:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:51:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:51:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:51:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:44.225378+00:00 (in 14.999509 seconds)
quotatotalworker stdout | 2025-02-07 03:51:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:44 UTC)" (scheduled at 2025-02-07 03:51:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:51:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:51:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:51:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:51:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:51:29,235 [235] [DEBUG] [app] Starting request: urn:request:1bf86db3-bc39-45e6-a02d-a62d57b9914c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:29,235 [238] [DEBUG] [app] Starting request: urn:request:4ec3452c-fc06-417d-a4a7-8cde9e877486 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:51:29,239 [243] [DEBUG] [app] Starting request: urn:request:4e949e11-1c92-407c-80d3-d6c6138acbda (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:29,239 [246] [DEBUG] [app] Starting request: urn:request:1f7b3f3e-c110-463c-b93f-b8e055c87b85 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:29,239 [246] [DEBUG] [app] Ending request: urn:request:1f7b3f3e-c110-463c-b93f-b8e055c87b85 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1f7b3f3e-c110-463c-b93f-b8e055c87b85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:51:29,239 [243] [DEBUG] [app] Ending request: urn:request:4e949e11-1c92-407c-80d3-d6c6138acbda (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e949e11-1c92-407c-80d3-d6c6138acbda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:51:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:51:29,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:29,243 [237] [DEBUG] [app] Starting request: urn:request:36870d64-c077-49df-8a4e-fd8b401b04c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:29,243 [237] [DEBUG] [app] Ending request: urn:request:36870d64-c077-49df-8a4e-fd8b401b04c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36870d64-c077-49df-8a4e-fd8b401b04c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:51:29,243 [236] [DEBUG] [app] Starting request: urn:request:3f7bb7fd-caaf-4506-ab1b-f349e9c22a6b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:29,244 [236] [DEBUG] [app] Ending request: urn:request:3f7bb7fd-caaf-4506-ab1b-f349e9c22a6b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f7bb7fd-caaf-4506-ab1b-f349e9c22a6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:51:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:51:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:29,255 [238] [DEBUG] [app] Ending request: urn:request:4ec3452c-fc06-417d-a4a7-8cde9e877486 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ec3452c-fc06-417d-a4a7-8cde9e877486', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:51:29,255 [235] [DEBUG] [app] Ending request: urn:request:1bf86db3-bc39-45e6-a02d-a62d57b9914c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1bf86db3-bc39-45e6-a02d-a62d57b9914c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:51:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:51:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
queuecleanupworker stdout | 2025-02-07 03:51:29,350 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:51:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:51:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:32.625945+00:00 (in 1.004399 seconds)
namespacegcworker stdout | 2025-02-07 03:51:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:31 UTC)" (scheduled at 2025-02-07 03:51:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:51:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:51:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 31, 621819), True, datetime.datetime(2025, 2, 7, 3, 51, 31, 621819), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:51:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:51:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:51:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:51:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:51:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:13.624301+00:00 (in 40.997902 seconds)
namespacegcworker stdout | 2025-02-07 03:51:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:32 UTC)" (scheduled at 2025-02-07 03:51:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:51:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:51:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:51:33,682 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:51:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:51:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:03.919286+00:00 (in 29.999532 seconds)
buildlogsarchiver stdout | 2025-02-07 03:51:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:03 UTC)" (scheduled at 2025-02-07 03:51:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:51:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 51, 33, 920040), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:51:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:51:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:51:33,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:51:34,596 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:56.713547+00:00 (in 18.997188 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:37 UTC)" (scheduled at 2025-02-07 03:51:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:37 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:51:38,727 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:51:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:51:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:48.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 03:51:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:48 UTC)" (scheduled at 2025-02-07 03:51:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:51:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:51:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 38, 732785), True, datetime.datetime(2025, 2, 7, 3, 51, 38, 732785), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:51:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:51:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:51:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:48 UTC)" executed successfully
builder stdout | 2025-02-07 03:51:39,328 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:51:39,865 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:51:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:51:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:54.423055+00:00 (in 13.998349 seconds)
gcworker stdout | 2025-02-07 03:51:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:10 UTC)" (scheduled at 2025-02-07 03:51:40.424250+00:00)
gcworker stdout | 2025-02-07 03:51:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:51:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900000430, None, 1, 0])
gcworker stdout | 2025-02-07 03:51:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:51:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:51:41,043 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:51:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:51:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:11.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 03:51:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:11 UTC)" (scheduled at 2025-02-07 03:51:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:51:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896701836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:51:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:51:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:51:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:51:41,936 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:51:42,056 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:51:43,082 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:43,393 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:51:44,109 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:51:44,121 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:51:44,151 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:51:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:51:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:59.225378+00:00 (in 14.999513 seconds)
quotatotalworker stdout | 2025-02-07 03:51:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:59 UTC)" (scheduled at 2025-02-07 03:51:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:51:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:51:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:51:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:51:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:51:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:51:44,236 [235] [DEBUG] [app] Starting request: urn:request:d1033af0-20e3-42ed-aa7d-d06e6d996c65 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:44,236 [238] [DEBUG] [app] Starting request: urn:request:032151a8-3157-42d6-95f5-1683da0c587e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:51:44,239 [239] [DEBUG] [app] Starting request: urn:request:08f3739a-4e13-45fc-98c0-5f3f59c24ae0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:44,239 [245] [DEBUG] [app] Starting request: urn:request:5d8424d4-b856-446a-bf76-28cd98bdb2d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:44,239 [239] [DEBUG] [app] Ending request: urn:request:08f3739a-4e13-45fc-98c0-5f3f59c24ae0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08f3739a-4e13-45fc-98c0-5f3f59c24ae0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:51:44,239 [245] [DEBUG] [app] Ending request: urn:request:5d8424d4-b856-446a-bf76-28cd98bdb2d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d8424d4-b856-446a-bf76-28cd98bdb2d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:51:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:51:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:44,242 [235] [DEBUG] [app] Starting request: urn:request:e6d2bcb7-2ce2-4ec7-91bd-209c94003802 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:44,242 [237] [DEBUG] [app] Starting request: urn:request:892aeb76-3296-4b3e-afa2-c71a4161be2c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:44,242 [235] [DEBUG] [app] Ending request: urn:request:e6d2bcb7-2ce2-4ec7-91bd-209c94003802 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6d2bcb7-2ce2-4ec7-91bd-209c94003802', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:51:44,242 [237] [DEBUG] [app] Ending request: urn:request:892aeb76-3296-4b3e-afa2-c71a4161be2c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:892aeb76-3296-4b3e-afa2-c71a4161be2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:51:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:51:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:51:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:44,253 [235] [DEBUG] [app] Ending request: urn:request:d1033af0-20e3-42ed-aa7d-d06e6d996c65 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1033af0-20e3-42ed-aa7d-d06e6d996c65', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:51:44,254 [238] [DEBUG] [app] Ending request: urn:request:032151a8-3157-42d6-95f5-1683da0c587e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:032151a8-3157-42d6-95f5-1683da0c587e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:51:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:51:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositorygcworker stdout | 2025-02-07 03:51:44,618 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:51:45,030 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:51:45,146 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:51:45,171 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:51:46,132 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:51:46,136 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:51:46,142 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:51:46,145 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:51:46,147 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:51:46,810 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:51:47,122 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:51:47,161 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:51:47,164 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:51:47,167 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:51:47,352 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:51:48,180 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:51:48,185 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:51:48,189 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:51:48,193 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:51:48,198 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:51:48,201 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:51:48,204 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:51:48,206 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:51:48,209 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:51:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:51:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:50.814677+00:00 (in 2.082065 seconds)
notificationworker stdout | 2025-02-07 03:51:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:58 UTC)" (scheduled at 2025-02-07 03:51:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:51:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:51:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 48, 732824), True, datetime.datetime(2025, 2, 7, 3, 51, 48, 732824), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:51:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:51:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:51:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:51:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:51:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:51:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:51:58.732147+00:00 (in 7.916998 seconds)
notificationworker stdout | 2025-02-07 03:51:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:50 UTC)" (scheduled at 2025-02-07 03:51:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:51:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:51:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:51:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:51:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:10.424250+00:00 (in 16.000745 seconds)
gcworker stdout | 2025-02-07 03:51:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:24 UTC)" (scheduled at 2025-02-07 03:51:54.423055+00:00)
gcworker stdout | 2025-02-07 03:51:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:51:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:51:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:51:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:56.118855+00:00 (in 59.999520 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:51:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:56 UTC)" (scheduled at 2025-02-07 03:51:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:51:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:51:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:51:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:51:56,356 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:37.715878+00:00 (in 41.001875 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:56 UTC)" (scheduled at 2025-02-07 03:51:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:51:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:51:58,304 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:51:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:51:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:08.732147+00:00 (in 9.999489 seconds)
notificationworker stdout | 2025-02-07 03:51:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:08 UTC)" (scheduled at 2025-02-07 03:51:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:51:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:51:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 51, 58, 732979), True, datetime.datetime(2025, 2, 7, 3, 51, 58, 732979), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:51:58,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:51:58,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:51:58,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:51:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:51:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:14.225378+00:00 (in 14.999494 seconds)
quotatotalworker stdout | 2025-02-07 03:51:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:14 UTC)" (scheduled at 2025-02-07 03:51:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:51:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:51:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:51:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:51:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:51:59,235 [237] [DEBUG] [app] Starting request: urn:request:b84b15fc-ca96-4592-959a-9273cebabeec (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:59,235 [238] [DEBUG] [app] Starting request: urn:request:c4dbaf46-11aa-450e-9d75-76c4b66bc60d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:51:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:51:59,238 [246] [DEBUG] [app] Starting request: urn:request:74a8048b-152a-4e1a-bb6f-29270069b745 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:59,238 [245] [DEBUG] [app] Starting request: urn:request:7672243e-7c4b-4d27-bd1b-8b6d0d0ac61d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:51:59,238 [245] [DEBUG] [app] Ending request: urn:request:7672243e-7c4b-4d27-bd1b-8b6d0d0ac61d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7672243e-7c4b-4d27-bd1b-8b6d0d0ac61d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:51:59,238 [246] [DEBUG] [app] Ending request: urn:request:74a8048b-152a-4e1a-bb6f-29270069b745 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:74a8048b-152a-4e1a-bb6f-29270069b745', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:51:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:51:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:51:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:51:59,241 [237] [DEBUG] [app] Starting request: urn:request:250013a1-7eaf-4796-b71d-4afea552221e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:59,241 [237] [DEBUG] [app] Ending request: urn:request:250013a1-7eaf-4796-b71d-4afea552221e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:250013a1-7eaf-4796-b71d-4afea552221e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:51:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:51:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:59,242 [236] [DEBUG] [app] Starting request: urn:request:40afae20-f323-4795-8abc-bc9dea5498f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:51:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:59,242 [236] [DEBUG] [app] Ending request: urn:request:40afae20-f323-4795-8abc-bc9dea5498f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:40afae20-f323-4795-8abc-bc9dea5498f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:51:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:51:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:51:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:51:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:51:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:51:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:51:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:51:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:51:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:51:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:51:59,253 [237] [DEBUG] [app] Ending request: urn:request:b84b15fc-ca96-4592-959a-9273cebabeec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b84b15fc-ca96-4592-959a-9273cebabeec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:51:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:51:59,254 [238] [DEBUG] [app] Ending request: urn:request:c4dbaf46-11aa-450e-9d75-76c4b66bc60d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4dbaf46-11aa-450e-9d75-76c4b66bc60d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:51:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:51:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:51:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:51:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 03:51:59,359 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:51:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:51:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:04.936688+00:00 (in 4.998311 seconds)
exportactionlogsworker stdout | 2025-02-07 03:51:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:59 UTC)" (scheduled at 2025-02-07 03:51:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:51:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:51:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:52:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:52:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:52:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:23.420886+00:00 (in 20.001290 seconds)
repositorygcworker stdout | 2025-02-07 03:52:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:03 UTC)" (scheduled at 2025-02-07 03:52:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:52:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:52:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 3, 419929), True, datetime.datetime(2025, 2, 7, 3, 52, 3, 419929), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:52:03,428 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:52:03,428 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:52:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:52:03,718 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:52:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:52:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:33.919286+00:00 (in 29.999478 seconds)
buildlogsarchiver stdout | 2025-02-07 03:52:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:33 UTC)" (scheduled at 2025-02-07 03:52:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:52:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 52, 3, 920102), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:52:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:52:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:52:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:52:04,632 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:52:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:52:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:59.937899+00:00 (in 55.000741 seconds)
exportactionlogsworker stdout | 2025-02-07 03:52:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:04 UTC)" (scheduled at 2025-02-07 03:52:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:52:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:52:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 4, 937472), True, datetime.datetime(2025, 2, 7, 3, 52, 4, 937472), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:52:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:52:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:52:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:52:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:18.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 03:52:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:18 UTC)" (scheduled at 2025-02-07 03:52:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:52:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:52:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 8, 732886), True, datetime.datetime(2025, 2, 7, 3, 52, 8, 732886), 0, 'notification/%', 50, 1, 0])
quotatotalworker stdout | 2025-02-07 03:52:08,734 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
notificationworker stdout | 2025-02-07 03:52:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:52:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:52:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:18 UTC)" executed successfully
builder stdout | 2025-02-07 03:52:09,365 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:52:09,901 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:52:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:52:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:24.423055+00:00 (in 13.998332 seconds)
gcworker stdout | 2025-02-07 03:52:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:40 UTC)" (scheduled at 2025-02-07 03:52:10.424250+00:00)
gcworker stdout | 2025-02-07 03:52:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:52:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900030431, None, 1, 0])
gcworker stdout | 2025-02-07 03:52:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:52:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:52:11,079 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:52:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:52:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:41.831521+00:00 (in 29.999541 seconds)
autopruneworker stdout | 2025-02-07 03:52:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:41 UTC)" (scheduled at 2025-02-07 03:52:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:52:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896731836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:52:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:52:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:52:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:52:11,973 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:52:12,092 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:52:13,108 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:13,429 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
namespacegcworker stdout | 2025-02-07 03:52:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:52:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:31.621131+00:00 (in 17.996394 seconds)
namespacegcworker stdout | 2025-02-07 03:52:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:57:13 UTC)" (scheduled at 2025-02-07 03:52:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:52:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 52, 13, 624986), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:52:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 52, 13, 624986), True, datetime.datetime(2025, 2, 7, 3, 52, 13, 624986), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:52:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 52, 13, 624986), True, datetime.datetime(2025, 2, 7, 3, 52, 13, 624986), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 52, 13, 624986), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:52:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:52:13,634 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:57:13 UTC)" executed successfully
servicekey stdout | 2025-02-07 03:52:14,145 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:52:14,156 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:52:14,187 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:52:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:52:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:29.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 03:52:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:29 UTC)" (scheduled at 2025-02-07 03:52:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:52:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:52:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:52:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:52:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:52:14,235 [238] [DEBUG] [app] Starting request: urn:request:744d8d31-33c5-48f8-b372-481440dc3044 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:14,235 [235] [DEBUG] [app] Starting request: urn:request:cf82b431-cc56-4c60-a5eb-8f0325401439 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:52:14,238 [239] [DEBUG] [app] Starting request: urn:request:dc12be39-9ae9-48af-833c-466a88ad6e91 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:14,238 [246] [DEBUG] [app] Starting request: urn:request:f4232fa1-5324-4afc-8d8b-f39556556eed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:14,239 [239] [DEBUG] [app] Ending request: urn:request:dc12be39-9ae9-48af-833c-466a88ad6e91 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc12be39-9ae9-48af-833c-466a88ad6e91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:52:14,239 [246] [DEBUG] [app] Ending request: urn:request:f4232fa1-5324-4afc-8d8b-f39556556eed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4232fa1-5324-4afc-8d8b-f39556556eed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:52:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:52:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:52:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:14,241 [235] [DEBUG] [app] Starting request: urn:request:f2877498-edb3-437e-8c5a-5b26b58617ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:14,242 [235] [DEBUG] [app] Ending request: urn:request:f2877498-edb3-437e-8c5a-5b26b58617ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2877498-edb3-437e-8c5a-5b26b58617ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:52:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:14,242 [237] [DEBUG] [app] Starting request: urn:request:38e96f94-557a-4648-9f8e-10eeb096a811 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:52:14,243 [237] [DEBUG] [app] Ending request: urn:request:38e96f94-557a-4648-9f8e-10eeb096a811 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:38e96f94-557a-4648-9f8e-10eeb096a811', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:52:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:52:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:52:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:14,253 [235] [DEBUG] [app] Ending request: urn:request:cf82b431-cc56-4c60-a5eb-8f0325401439 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf82b431-cc56-4c60-a5eb-8f0325401439', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:52:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:52:14,254 [238] [DEBUG] [app] Ending request: urn:request:744d8d31-33c5-48f8-b372-481440dc3044 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:744d8d31-33c5-48f8-b372-481440dc3044', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:52:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositorygcworker stdout | 2025-02-07 03:52:14,650 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:52:15,066 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:52:15,180 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:52:15,183 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:52:16,140 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:52:16,144 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:52:16,148 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:52:16,151 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:52:16,154 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:52:16,823 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:52:17,140 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:52:17,170 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:52:17,173 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:52:17,176 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:52:17,388 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:52:18,191 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:52:18,195 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:52:18,199 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:52:18,201 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:52:18,206 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:52:18,208 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:52:18,211 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:52:18,213 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:52:18,219 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:52:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:28.732147+00:00 (in 9.999402 seconds)
notificationworker stdout | 2025-02-07 03:52:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:28 UTC)" (scheduled at 2025-02-07 03:52:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:52:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:52:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 18, 733030), True, datetime.datetime(2025, 2, 7, 3, 52, 18, 733030), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:52:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:52:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:52:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:52:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:52:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:03.419110+00:00 (in 39.997725 seconds)
repositorygcworker stdout | 2025-02-07 03:52:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:23 UTC)" (scheduled at 2025-02-07 03:52:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:52:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:52:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:52:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:52:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:40.424250+00:00 (in 16.000746 seconds)
gcworker stdout | 2025-02-07 03:52:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:54 UTC)" (scheduled at 2025-02-07 03:52:24.423055+00:00)
gcworker stdout | 2025-02-07 03:52:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:52:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:52:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:52:26,393 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:52:28,331 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:52:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:33.814166+00:00 (in 5.081487 seconds)
notificationworker stdout | 2025-02-07 03:52:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:38 UTC)" (scheduled at 2025-02-07 03:52:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:52:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:52:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 28, 732993), True, datetime.datetime(2025, 2, 7, 3, 52, 28, 732993), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:52:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:52:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:52:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:52:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:52:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:44.225378+00:00 (in 14.999579 seconds)
quotatotalworker stdout | 2025-02-07 03:52:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:44 UTC)" (scheduled at 2025-02-07 03:52:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:52:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:52:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:52:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:52:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:52:29,236 [236] [DEBUG] [app] Starting request: urn:request:a9dec883-97ce-4303-8d8b-0be5788f2cd1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:29,236 [238] [DEBUG] [app] Starting request: urn:request:2f14dfeb-714f-404f-b2e2-1f871627d336 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:52:29,238 [239] [DEBUG] [app] Starting request: urn:request:a52a8c54-c4a6-4515-81d0-6123a2f2cac7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:29,238 [246] [DEBUG] [app] Starting request: urn:request:f3121970-b383-4697-b29d-c1c28781c7de (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:29,239 [239] [DEBUG] [app] Ending request: urn:request:a52a8c54-c4a6-4515-81d0-6123a2f2cac7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a52a8c54-c4a6-4515-81d0-6123a2f2cac7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:52:29,239 [246] [DEBUG] [app] Ending request: urn:request:f3121970-b383-4697-b29d-c1c28781c7de (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3121970-b383-4697-b29d-c1c28781c7de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:52:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:52:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:52:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:29,242 [237] [DEBUG] [app] Starting request: urn:request:320a5d6c-5360-4b9f-b263-400cc90ec331 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:29,242 [237] [DEBUG] [app] Ending request: urn:request:320a5d6c-5360-4b9f-b263-400cc90ec331 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:320a5d6c-5360-4b9f-b263-400cc90ec331', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:52:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:29,243 [235] [DEBUG] [app] Starting request: urn:request:6b0cacb1-4933-43f2-bc71-44866cbac281 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:29,243 [235] [DEBUG] [app] Ending request: urn:request:6b0cacb1-4933-43f2-bc71-44866cbac281 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b0cacb1-4933-43f2-bc71-44866cbac281', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:52:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:52:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:52:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:52:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:29,254 [238] [DEBUG] [app] Ending request: urn:request:2f14dfeb-714f-404f-b2e2-1f871627d336 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f14dfeb-714f-404f-b2e2-1f871627d336', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:52:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:52:29,255 [236] [DEBUG] [app] Ending request: urn:request:a9dec883-97ce-4303-8d8b-0be5788f2cd1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a9dec883-97ce-4303-8d8b-0be5788f2cd1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:52:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 03:52:29,395 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:52:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:52:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:32.625945+00:00 (in 1.004372 seconds)
namespacegcworker stdout | 2025-02-07 03:52:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:31 UTC)" (scheduled at 2025-02-07 03:52:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:52:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:52:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 31, 621889), True, datetime.datetime(2025, 2, 7, 3, 52, 31, 621889), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:52:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:52:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:52:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:52:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:52:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:31.621131+00:00 (in 58.994739 seconds)
namespacegcworker stdout | 2025-02-07 03:52:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:32 UTC)" (scheduled at 2025-02-07 03:52:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:52:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:52:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:52:33,747 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 03:52:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:38.732147+00:00 (in 4.917491 seconds)
notificationworker stdout | 2025-02-07 03:52:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:57:33 UTC)" (scheduled at 2025-02-07 03:52:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:52:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 52, 33, 814840), 'notification/%'])
notificationworker stdout | 2025-02-07 03:52:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 52, 33, 814840), True, datetime.datetime(2025, 2, 7, 3, 52, 33, 814840), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:52:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 52, 33, 814840), True, datetime.datetime(2025, 2, 7, 3, 52, 33, 814840), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 52, 33, 814840), 'notification/%'])
notificationworker stdout | 2025-02-07 03:52:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:52:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:57:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:52:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:52:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:03.919286+00:00 (in 29.999477 seconds)
buildlogsarchiver stdout | 2025-02-07 03:52:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:03 UTC)" (scheduled at 2025-02-07 03:52:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:52:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 52, 33, 920079), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:52:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:52:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:52:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:52:34,668 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:56.713547+00:00 (in 18.997196 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:37 UTC)" (scheduled at 2025-02-07 03:52:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:52:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:48.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 03:52:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:48 UTC)" (scheduled at 2025-02-07 03:52:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:52:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:52:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 38, 732799), True, datetime.datetime(2025, 2, 7, 3, 52, 38, 732799), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:52:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:52:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:52:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:52:38,770 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:52:39,389 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:52:39,937 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:52:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:52:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:54.423055+00:00 (in 13.998369 seconds)
gcworker stdout | 2025-02-07 03:52:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:10 UTC)" (scheduled at 2025-02-07 03:52:40.424250+00:00)
gcworker stdout | 2025-02-07 03:52:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:52:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900060431, None, 1, 0])
gcworker stdout | 2025-02-07 03:52:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:52:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:52:41,089 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:52:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:52:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:11.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 03:52:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:11 UTC)" (scheduled at 2025-02-07 03:52:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:52:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896761836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:52:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:52:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:52:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:52:42,009 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:52:42,112 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:52:43,144 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:43,450 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:52:44,178 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:52:44,192 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:52:44,223 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
quotatotalworker stdout | 2025-02-07 03:52:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:52:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:59.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 03:52:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:59 UTC)" (scheduled at 2025-02-07 03:52:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:52:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:52:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:52:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:52:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:52:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:52:44,235 [238] [DEBUG] [app] Starting request: urn:request:620310a2-9724-41d9-8c96-6bc324a32cad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:44,235 [235] [DEBUG] [app] Starting request: urn:request:fd7bf272-4fd9-47b2-bd74-a1ffc45d9390 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:52:44,238 [239] [DEBUG] [app] Starting request: urn:request:b9aa7031-c595-4f5b-ae36-ca8b5573a50e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:44,238 [245] [DEBUG] [app] Starting request: urn:request:d119ef83-d694-4bec-8f89-21bce613a226 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:44,239 [245] [DEBUG] [app] Ending request: urn:request:d119ef83-d694-4bec-8f89-21bce613a226 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d119ef83-d694-4bec-8f89-21bce613a226', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:52:44,239 [239] [DEBUG] [app] Ending request: urn:request:b9aa7031-c595-4f5b-ae36-ca8b5573a50e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9aa7031-c595-4f5b-ae36-ca8b5573a50e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:52:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:52:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:52:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:44,241 [235] [DEBUG] [app] Starting request: urn:request:3c4e901b-a53c-4228-a456-868de902372c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:44,242 [235] [DEBUG] [app] Ending request: urn:request:3c4e901b-a53c-4228-a456-868de902372c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c4e901b-a53c-4228-a456-868de902372c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:52:44,242 [237] [DEBUG] [app] Starting request: urn:request:55c6474d-2ff3-46c4-8dec-71d5fc532f27 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:44,242 [237] [DEBUG] [app] Ending request: urn:request:55c6474d-2ff3-46c4-8dec-71d5fc532f27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55c6474d-2ff3-46c4-8dec-71d5fc532f27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:52:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:52:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:52:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:44,253 [235] [DEBUG] [app] Ending request: urn:request:fd7bf272-4fd9-47b2-bd74-a1ffc45d9390 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd7bf272-4fd9-47b2-bd74-a1ffc45d9390', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:52:44,254 [238] [DEBUG] [app] Ending request: urn:request:620310a2-9724-41d9-8c96-6bc324a32cad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:620310a2-9724-41d9-8c96-6bc324a32cad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:52:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:52:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositorygcworker stdout | 2025-02-07 03:52:44,669 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:52:45,103 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:52:45,206 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:52:45,216 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:52:46,149 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:52:46,151 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:52:46,155 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:52:46,158 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:52:46,162 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:52:46,832 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 03:52:47,168 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-secscan stdout | 2025-02-07 03:52:47,179 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:52:47,184 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:52:47,187 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
blobuploadcleanupworker stdout | 2025-02-07 03:52:47,425 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:52:48,201 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:52:48,205 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:52:48,208 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:52:48,211 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:52:48,214 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:52:48,216 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:52:48,223 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:52:48,226 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 03:52:48,228 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 03:52:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:50.814677+00:00 (in 2.082083 seconds)
notificationworker stdout | 2025-02-07 03:52:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:58 UTC)" (scheduled at 2025-02-07 03:52:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:52:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:52:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 48, 732873), True, datetime.datetime(2025, 2, 7, 3, 52, 48, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:52:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:52:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:52:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:52:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:52:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:52:58.732147+00:00 (in 7.917006 seconds)
notificationworker stdout | 2025-02-07 03:52:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:50 UTC)" (scheduled at 2025-02-07 03:52:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:52:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:52:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:52:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:52:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:10.424250+00:00 (in 16.000720 seconds)
gcworker stdout | 2025-02-07 03:52:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:24 UTC)" (scheduled at 2025-02-07 03:52:54.423055+00:00)
gcworker stdout | 2025-02-07 03:52:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:52:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:52:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:52:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:56.118855+00:00 (in 59.999515 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:52:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:56 UTC)" (scheduled at 2025-02-07 03:52:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:52:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:52:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:52:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:52:56,418 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:37.715878+00:00 (in 41.001849 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:56 UTC)" (scheduled at 2025-02-07 03:52:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:52:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:52:58,367 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:52:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:52:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:08.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 03:52:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:08 UTC)" (scheduled at 2025-02-07 03:52:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:52:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:52:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 52, 58, 732869), True, datetime.datetime(2025, 2, 7, 3, 52, 58, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:52:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:52:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:52:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:52:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:52:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:14.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 03:52:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:14 UTC)" (scheduled at 2025-02-07 03:52:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:52:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:52:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:52:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:52:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:52:59,236 [237] [DEBUG] [app] Starting request: urn:request:41b14daf-e63f-42b3-85ca-b43a153308b7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:59,236 [235] [DEBUG] [app] Starting request: urn:request:0e03c805-ce01-49c3-9375-906592a88f4a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:52:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:59,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:52:59,240 [243] [DEBUG] [app] Starting request: urn:request:7692bc96-a730-44a7-b36b-7e5572a9ce60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:59,240 [239] [DEBUG] [app] Starting request: urn:request:981082af-71f3-470f-959a-97d98900681f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:52:59,240 [243] [DEBUG] [app] Ending request: urn:request:7692bc96-a730-44a7-b36b-7e5572a9ce60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7692bc96-a730-44a7-b36b-7e5572a9ce60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:52:59,240 [239] [DEBUG] [app] Ending request: urn:request:981082af-71f3-470f-959a-97d98900681f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:981082af-71f3-470f-959a-97d98900681f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:52:59,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:52:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:52:59,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:52:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:59,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:52:59,244 [238] [DEBUG] [app] Starting request: urn:request:167ceb09-29d0-44ca-8ec3-e7effbcf863b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:59,244 [236] [DEBUG] [app] Starting request: urn:request:ebaea51a-ba3c-4867-a0b6-fdea0d6c925d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:52:59,244 [238] [DEBUG] [app] Ending request: urn:request:167ceb09-29d0-44ca-8ec3-e7effbcf863b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:167ceb09-29d0-44ca-8ec3-e7effbcf863b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:52:59,244 [236] [DEBUG] [app] Ending request: urn:request:ebaea51a-ba3c-4867-a0b6-fdea0d6c925d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ebaea51a-ba3c-4867-a0b6-fdea0d6c925d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:52:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:52:59,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:52:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:52:59,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:52:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:52:59,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:59,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:52:59,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:52:59,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:52:59,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:59,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:52:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:52:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:52:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:52:59,256 [237] [DEBUG] [app] Ending request: urn:request:41b14daf-e63f-42b3-85ca-b43a153308b7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:41b14daf-e63f-42b3-85ca-b43a153308b7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:59,256 [235] [DEBUG] [app] Ending request: urn:request:0e03c805-ce01-49c3-9375-906592a88f4a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e03c805-ce01-49c3-9375-906592a88f4a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:52:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:52:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:52:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:52:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:52:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:52:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
queuecleanupworker stdout | 2025-02-07 03:52:59,401 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:52:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:52:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:04.936688+00:00 (in 4.998281 seconds)
exportactionlogsworker stdout | 2025-02-07 03:52:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:59 UTC)" (scheduled at 2025-02-07 03:52:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:52:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:52:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:53:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:53:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:53:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:23.420886+00:00 (in 20.001294 seconds)
repositorygcworker stdout | 2025-02-07 03:53:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:03 UTC)" (scheduled at 2025-02-07 03:53:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:53:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:53:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 3, 419794), True, datetime.datetime(2025, 2, 7, 3, 53, 3, 419794), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:53:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:53:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:53:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:53:03,774 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:53:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:53:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:33.919286+00:00 (in 29.999543 seconds)
buildlogsarchiver stdout | 2025-02-07 03:53:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:33 UTC)" (scheduled at 2025-02-07 03:53:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:53:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 53, 3, 919989), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:53:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:53:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:53:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:53:04,705 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:53:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:53:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:59.937899+00:00 (in 55.000767 seconds)
exportactionlogsworker stdout | 2025-02-07 03:53:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:04 UTC)" (scheduled at 2025-02-07 03:53:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:53:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:53:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 4, 937448), True, datetime.datetime(2025, 2, 7, 3, 53, 4, 937448), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:53:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:53:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:53:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:53:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:53:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:18.732147+00:00 (in 9.999573 seconds)
notificationworker stdout | 2025-02-07 03:53:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:18 UTC)" (scheduled at 2025-02-07 03:53:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:53:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:53:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 8, 732856), True, datetime.datetime(2025, 2, 7, 3, 53, 8, 732856), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:53:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:53:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:53:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:53:08,807 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:53:09,425 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:53:09,973 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:53:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:53:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:24.423055+00:00 (in 13.998369 seconds)
gcworker stdout | 2025-02-07 03:53:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:40 UTC)" (scheduled at 2025-02-07 03:53:10.424250+00:00)
gcworker stdout | 2025-02-07 03:53:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:53:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900090431, None, 1, 0])
gcworker stdout | 2025-02-07 03:53:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:53:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:53:11,099 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:53:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:53:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 03:53:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:41 UTC)" (scheduled at 2025-02-07 03:53:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:53:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896791836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:53:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:53:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:53:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:53:12,046 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:53:12,138 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:53:13,180 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:13,485 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 03:53:14,214 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
quotatotalworker stdout | 2025-02-07 03:53:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:53:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:29.225378+00:00 (in 14.999584 seconds)
quotatotalworker stdout | 2025-02-07 03:53:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:29 UTC)" (scheduled at 2025-02-07 03:53:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:53:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:53:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
globalpromstats stdout | 2025-02-07 03:53:14,227 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
quotatotalworker stdout | 2025-02-07 03:53:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:53:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:53:14,235 [235] [DEBUG] [app] Starting request: urn:request:4b588d80-a161-45d8-85d0-b616c72dbbdb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:14,235 [237] [DEBUG] [app] Starting request: urn:request:0f9b6941-a8e8-454f-a5ad-14f65a164c4c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:53:14,238 [239] [DEBUG] [app] Starting request: urn:request:a264f97b-e4fa-4150-b6d6-922512cd9693 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:14,238 [246] [DEBUG] [app] Starting request: urn:request:678d4b77-1ec7-45a7-bbb9-7c31dbb43967 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:14,238 [239] [DEBUG] [app] Ending request: urn:request:a264f97b-e4fa-4150-b6d6-922512cd9693 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a264f97b-e4fa-4150-b6d6-922512cd9693', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:53:14,239 [246] [DEBUG] [app] Ending request: urn:request:678d4b77-1ec7-45a7-bbb9-7c31dbb43967 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:678d4b77-1ec7-45a7-bbb9-7c31dbb43967', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:53:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:53:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:53:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:14,241 [238] [DEBUG] [app] Starting request: urn:request:59daa606-020f-4e64-892f-61662bcf4068 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:53:14,242 [238] [DEBUG] [app] Ending request: urn:request:59daa606-020f-4e64-892f-61662bcf4068 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59daa606-020f-4e64-892f-61662bcf4068', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:53:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:53:14,242 [235] [DEBUG] [app] Starting request: urn:request:49457e1b-70f4-4030-b187-b85148072698 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:53:14,242 [235] [DEBUG] [app] Ending request: urn:request:49457e1b-70f4-4030-b187-b85148072698 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:49457e1b-70f4-4030-b187-b85148072698', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:53:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:14,254 [235] [DEBUG] [app] Ending request: urn:request:4b588d80-a161-45d8-85d0-b616c72dbbdb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4b588d80-a161-45d8-85d0-b616c72dbbdb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:53:14,254 [237] [DEBUG] [app] Ending request: urn:request:0f9b6941-a8e8-454f-a5ad-14f65a164c4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f9b6941-a8e8-454f-a5ad-14f65a164c4c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:53:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:53:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 03:53:14,259 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:53:14,705 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:53:15,139 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:53:15,242 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:53:15,247 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:53:16,157 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:53:16,159 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:53:16,162 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:53:16,164 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:53:16,168 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:53:16,868 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:53:17,189 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:53:17,192 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:53:17,196 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:53:17,202 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:53:17,460 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:53:18,210 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:53:18,216 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:53:18,219 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:53:18,223 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:53:18,226 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:53:18,228 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:53:18,234 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:53:18,237 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:53:18,240 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:53:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:53:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:28.732147+00:00 (in 9.999520 seconds)
notificationworker stdout | 2025-02-07 03:53:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:28 UTC)" (scheduled at 2025-02-07 03:53:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:53:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:53:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 18, 732836), True, datetime.datetime(2025, 2, 7, 3, 53, 18, 732836), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:53:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:53:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:53:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:53:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:53:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:32.420707+00:00 (in 8.999339 seconds)
repositorygcworker stdout | 2025-02-07 03:53:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:23 UTC)" (scheduled at 2025-02-07 03:53:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:53:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:53:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:53:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:53:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:40.424250+00:00 (in 16.000736 seconds)
gcworker stdout | 2025-02-07 03:53:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:54 UTC)" (scheduled at 2025-02-07 03:53:24.423055+00:00)
gcworker stdout | 2025-02-07 03:53:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:53:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:53:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:53:26,433 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:53:28,390 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:53:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:53:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:38.732147+00:00 (in 9.999547 seconds)
notificationworker stdout | 2025-02-07 03:53:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:38 UTC)" (scheduled at 2025-02-07 03:53:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:53:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:53:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 28, 732815), True, datetime.datetime(2025, 2, 7, 3, 53, 28, 732815), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:53:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:53:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:53:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:53:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:53:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:44.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 03:53:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:44 UTC)" (scheduled at 2025-02-07 03:53:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:53:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:53:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:53:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:53:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:53:29,235 [235] [DEBUG] [app] Starting request: urn:request:37eb058c-c012-4dfe-8471-4c9b90f8c164 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:29,235 [238] [DEBUG] [app] Starting request: urn:request:247b654b-0803-41cc-9953-928da7483aa0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:53:29,238 [246] [DEBUG] [app] Starting request: urn:request:1819fd29-3cf1-4ee6-98ba-51168386a660 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:29,238 [239] [DEBUG] [app] Starting request: urn:request:18f450be-cf92-4938-9e68-a4ebf137e15b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:29,238 [239] [DEBUG] [app] Ending request: urn:request:18f450be-cf92-4938-9e68-a4ebf137e15b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18f450be-cf92-4938-9e68-a4ebf137e15b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:53:29,238 [246] [DEBUG] [app] Ending request: urn:request:1819fd29-3cf1-4ee6-98ba-51168386a660 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1819fd29-3cf1-4ee6-98ba-51168386a660', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:53:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:53:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:29,241 [235] [DEBUG] [app] Starting request: urn:request:c8c2ebc6-2bae-4ec9-9245-425659b636fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:53:29,241 [235] [DEBUG] [app] Ending request: urn:request:c8c2ebc6-2bae-4ec9-9245-425659b636fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c8c2ebc6-2bae-4ec9-9245-425659b636fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:53:29,241 [237] [DEBUG] [app] Starting request: urn:request:fa707a91-044e-4638-9d16-36641c8016c6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:53:29,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:29,241 [237] [DEBUG] [app] Ending request: urn:request:fa707a91-044e-4638-9d16-36641c8016c6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa707a91-044e-4638-9d16-36641c8016c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:53:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:53:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:29,253 [238] [DEBUG] [app] Ending request: urn:request:247b654b-0803-41cc-9953-928da7483aa0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:247b654b-0803-41cc-9953-928da7483aa0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:29,253 [235] [DEBUG] [app] Ending request: urn:request:37eb058c-c012-4dfe-8471-4c9b90f8c164 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37eb058c-c012-4dfe-8471-4c9b90f8c164', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:53:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:53:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:53:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 03:53:29,436 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:53:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:53:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:32.625945+00:00 (in 1.004338 seconds)
namespacegcworker stdout | 2025-02-07 03:53:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:31 UTC)" (scheduled at 2025-02-07 03:53:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:53:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:53:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 31, 621890), True, datetime.datetime(2025, 2, 7, 3, 53, 31, 621890), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:53:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:53:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:53:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:53:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:53:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:03.419110+00:00 (in 30.997939 seconds)
repositorygcworker stdout | 2025-02-07 03:53:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:58:32 UTC)" (scheduled at 2025-02-07 03:53:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:53:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 53, 32, 421388), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:53:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 53, 32, 421388), True, datetime.datetime(2025, 2, 7, 3, 53, 32, 421388), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:53:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 53, 32, 421388), True, datetime.datetime(2025, 2, 7, 3, 53, 32, 421388), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 53, 32, 421388), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:53:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:53:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:58:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:53:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:53:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:31.621131+00:00 (in 58.994720 seconds)
namespacegcworker stdout | 2025-02-07 03:53:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:32 UTC)" (scheduled at 2025-02-07 03:53:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:53:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:53:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:53:33,804 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:53:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:53:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:03.919286+00:00 (in 29.999521 seconds)
buildlogsarchiver stdout | 2025-02-07 03:53:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:03 UTC)" (scheduled at 2025-02-07 03:53:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:53:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 53, 33, 920050), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:53:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:53:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:53:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:53:34,741 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:56.713547+00:00 (in 18.997213 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:37 UTC)" (scheduled at 2025-02-07 03:53:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:53:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:53:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:48.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 03:53:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:48 UTC)" (scheduled at 2025-02-07 03:53:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:53:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:53:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 38, 732797), True, datetime.datetime(2025, 2, 7, 3, 53, 38, 732797), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:53:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:53:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:53:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:53:38,832 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:53:39,436 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:53:40,010 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:53:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:53:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:54.423055+00:00 (in 13.998381 seconds)
gcworker stdout | 2025-02-07 03:53:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:10 UTC)" (scheduled at 2025-02-07 03:53:40.424250+00:00)
gcworker stdout | 2025-02-07 03:53:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:53:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900120431, None, 1, 0])
gcworker stdout | 2025-02-07 03:53:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:53:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:53:41,129 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:53:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:53:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:11.831521+00:00 (in 29.999525 seconds)
autopruneworker stdout | 2025-02-07 03:53:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:11 UTC)" (scheduled at 2025-02-07 03:53:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:53:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896821836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:53:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:53:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:53:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:53:42,079 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:53:42,174 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:53:43,216 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:43,506 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:53:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:53:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:59.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 03:53:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:59 UTC)" (scheduled at 2025-02-07 03:53:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:53:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:53:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:53:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:53:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:53:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:53:44,235 [238] [DEBUG] [app] Starting request: urn:request:08bbcc69-ec31-4035-bfe0-797d5b69c1a7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:44,235 [235] [DEBUG] [app] Starting request: urn:request:9e7e228f-d42a-4319-8fc5-37638a0cf926 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:53:44,238 [239] [DEBUG] [app] Starting request: urn:request:40c7331b-ab95-4843-a54f-ef835bef9f53 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:44,238 [246] [DEBUG] [app] Starting request: urn:request:3e61d914-8deb-4cfd-94fd-b29932a1532d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:44,239 [246] [DEBUG] [app] Ending request: urn:request:3e61d914-8deb-4cfd-94fd-b29932a1532d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e61d914-8deb-4cfd-94fd-b29932a1532d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:53:44,239 [239] [DEBUG] [app] Ending request: urn:request:40c7331b-ab95-4843-a54f-ef835bef9f53 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40c7331b-ab95-4843-a54f-ef835bef9f53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
servicekey stdout | 2025-02-07 03:53:44,239 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 03:53:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:53:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:53:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:44,241 [237] [DEBUG] [app] Starting request: urn:request:301391ce-7acc-4762-9e19-668dea5aa864 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:53:44,242 [237] [DEBUG] [app] Ending request: urn:request:301391ce-7acc-4762-9e19-668dea5aa864 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:301391ce-7acc-4762-9e19-668dea5aa864', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:53:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:44,242 [236] [DEBUG] [app] Starting request: urn:request:a1aa92e9-9e5b-4aa9-a5e7-984547e85788 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:53:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:44,242 [236] [DEBUG] [app] Ending request: urn:request:a1aa92e9-9e5b-4aa9-a5e7-984547e85788 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1aa92e9-9e5b-4aa9-a5e7-984547e85788', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:53:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:53:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:44,253 [238] [DEBUG] [app] Ending request: urn:request:08bbcc69-ec31-4035-bfe0-797d5b69c1a7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08bbcc69-ec31-4035-bfe0-797d5b69c1a7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:53:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:53:44,254 [235] [DEBUG] [app] Ending request: urn:request:9e7e228f-d42a-4319-8fc5-37638a0cf926 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e7e228f-d42a-4319-8fc5-37638a0cf926', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:53:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
globalpromstats stdout | 2025-02-07 03:53:44,262 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:53:44,295 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:53:44,741 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:53:45,169 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:53:45,279 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:53:45,281 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:53:46,165 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:53:46,169 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:53:46,171 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:53:46,175 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:53:46,177 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:53:46,901 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:53:47,197 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:53:47,200 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:53:47,204 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:53:47,238 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:53:47,479 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:53:48,218 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:53:48,228 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:53:48,231 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:53:48,233 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:53:48,236 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:53:48,239 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:53:48,242 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:53:48,245 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:53:48,249 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:53:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:53:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:50.814677+00:00 (in 2.082057 seconds)
notificationworker stdout | 2025-02-07 03:53:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:58 UTC)" (scheduled at 2025-02-07 03:53:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:53:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:53:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 48, 732915), True, datetime.datetime(2025, 2, 7, 3, 53, 48, 732915), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:53:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:53:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:53:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:53:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:53:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:53:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:53:58.732147+00:00 (in 7.916947 seconds)
notificationworker stdout | 2025-02-07 03:53:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:50 UTC)" (scheduled at 2025-02-07 03:53:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:53:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:53:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:53:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:53:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:10.424250+00:00 (in 16.000747 seconds)
gcworker stdout | 2025-02-07 03:53:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:24 UTC)" (scheduled at 2025-02-07 03:53:54.423055+00:00)
gcworker stdout | 2025-02-07 03:53:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:53:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:53:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:53:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:56.118855+00:00 (in 59.999464 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:53:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:56 UTC)" (scheduled at 2025-02-07 03:53:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:53:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:53:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:53:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:53:56,470 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:37.715878+00:00 (in 41.001852 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:56 UTC)" (scheduled at 2025-02-07 03:53:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:53:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:53:58,401 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:53:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:53:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:08.732147+00:00 (in 9.999508 seconds)
notificationworker stdout | 2025-02-07 03:53:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:08 UTC)" (scheduled at 2025-02-07 03:53:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:53:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:53:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 53, 58, 732947), True, datetime.datetime(2025, 2, 7, 3, 53, 58, 732947), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:53:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:53:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:53:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:53:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:53:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:14.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 03:53:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:14 UTC)" (scheduled at 2025-02-07 03:53:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:53:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:53:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:53:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:53:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:53:59,236 [235] [DEBUG] [app] Starting request: urn:request:b6114b12-85ca-440d-86a5-0f1c27290624 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:59,236 [238] [DEBUG] [app] Starting request: urn:request:434f2698-34a9-402e-8c11-5777c57e7331 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:53:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:59,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:53:59,240 [239] [DEBUG] [app] Starting request: urn:request:cddacc7e-bf91-4c81-ae04-4881ae4216c0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:59,240 [245] [DEBUG] [app] Starting request: urn:request:356b92fc-6951-4df7-869e-3fa67fc7fe9c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:53:59,240 [245] [DEBUG] [app] Ending request: urn:request:356b92fc-6951-4df7-869e-3fa67fc7fe9c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:356b92fc-6951-4df7-869e-3fa67fc7fe9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:53:59,240 [239] [DEBUG] [app] Ending request: urn:request:cddacc7e-bf91-4c81-ae04-4881ae4216c0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cddacc7e-bf91-4c81-ae04-4881ae4216c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:53:59,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:53:59,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:59,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:59,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:53:59,243 [235] [DEBUG] [app] Starting request: urn:request:6c3b74f0-8e27-40e1-bb50-edeff3ef0969 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:53:59,243 [238] [DEBUG] [app] Starting request: urn:request:4b6223c1-32a9-4623-bc7b-0cf45ae29bf2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:53:59,243 [235] [DEBUG] [app] Ending request: urn:request:6c3b74f0-8e27-40e1-bb50-edeff3ef0969 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6c3b74f0-8e27-40e1-bb50-edeff3ef0969', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:53:59,243 [238] [DEBUG] [app] Ending request: urn:request:4b6223c1-32a9-4623-bc7b-0cf45ae29bf2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b6223c1-32a9-4623-bc7b-0cf45ae29bf2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:53:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:53:59,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:53:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:53:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:53:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:53:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:53:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:53:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:53:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:53:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:53:59,255 [238] [DEBUG] [app] Ending request: urn:request:434f2698-34a9-402e-8c11-5777c57e7331 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:434f2698-34a9-402e-8c11-5777c57e7331', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:59,255 [235] [DEBUG] [app] Ending request: urn:request:b6114b12-85ca-440d-86a5-0f1c27290624 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6114b12-85ca-440d-86a5-0f1c27290624', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:53:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:53:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:53:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:53:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:53:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:53:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 03:53:59,472 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:53:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:53:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:04.936688+00:00 (in 4.998324 seconds)
exportactionlogsworker stdout | 2025-02-07 03:53:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:59 UTC)" (scheduled at 2025-02-07 03:53:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:53:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:53:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:54:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:54:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:54:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:23.420886+00:00 (in 20.001293 seconds)
repositorygcworker stdout | 2025-02-07 03:54:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:03 UTC)" (scheduled at 2025-02-07 03:54:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:54:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:54:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 3, 419802), True, datetime.datetime(2025, 2, 7, 3, 54, 3, 419802), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:54:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:54:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:54:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:54:03,822 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:54:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:54:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:33.919286+00:00 (in 29.999509 seconds)
buildlogsarchiver stdout | 2025-02-07 03:54:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:33 UTC)" (scheduled at 2025-02-07 03:54:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:54:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 54, 3, 920028), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:54:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:54:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:54:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:54:04,774 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:54:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:54:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:52.937648+00:00 (in 48.000526 seconds)
exportactionlogsworker stdout | 2025-02-07 03:54:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:04 UTC)" (scheduled at 2025-02-07 03:54:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:54:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:54:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 4, 937425), True, datetime.datetime(2025, 2, 7, 3, 54, 4, 937425), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:54:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:54:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:54:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:54:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:54:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:18.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 03:54:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:18 UTC)" (scheduled at 2025-02-07 03:54:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:54:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:54:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 8, 732940), True, datetime.datetime(2025, 2, 7, 3, 54, 8, 732940), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:54:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:54:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:54:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:54:08,843 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:54:09,447 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:54:10,020 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:54:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:54:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:24.423055+00:00 (in 13.998363 seconds)
gcworker stdout | 2025-02-07 03:54:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:40 UTC)" (scheduled at 2025-02-07 03:54:10.424250+00:00)
gcworker stdout | 2025-02-07 03:54:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:54:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900150430, None, 1, 0])
gcworker stdout | 2025-02-07 03:54:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:54:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:54:11,165 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:54:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:54:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:41.831521+00:00 (in 29.999547 seconds)
autopruneworker stdout | 2025-02-07 03:54:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:41 UTC)" (scheduled at 2025-02-07 03:54:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:54:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896851836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:54:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:54:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:54:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:54:12,090 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:54:12,210 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:54:13,244 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:13,533 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:54:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:54:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:29.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 03:54:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:29 UTC)" (scheduled at 2025-02-07 03:54:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:54:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:54:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:54:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:54:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:54:14,235 [238] [DEBUG] [app] Starting request: urn:request:6c89fffd-8cc5-4654-9456-5e0f1c398d04 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:14,235 [235] [DEBUG] [app] Starting request: urn:request:9bc654ae-431c-48a7-913b-eb7b8c77fa2c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:54:14,239 [246] [DEBUG] [app] Starting request: urn:request:b0a620f4-2e9b-4ae1-b091-a493ddbcd374 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:14,239 [245] [DEBUG] [app] Starting request: urn:request:1b6da490-9d3b-43ed-becc-c6eeb8672645 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:14,239 [246] [DEBUG] [app] Ending request: urn:request:b0a620f4-2e9b-4ae1-b091-a493ddbcd374 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b0a620f4-2e9b-4ae1-b091-a493ddbcd374', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:54:14,239 [245] [DEBUG] [app] Ending request: urn:request:1b6da490-9d3b-43ed-becc-c6eeb8672645 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b6da490-9d3b-43ed-becc-c6eeb8672645', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:54:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:54:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:54:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:14,242 [235] [DEBUG] [app] Starting request: urn:request:f9863b91-e075-4caf-b591-8048fd2c4144 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:14,243 [235] [DEBUG] [app] Ending request: urn:request:f9863b91-e075-4caf-b591-8048fd2c4144 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9863b91-e075-4caf-b591-8048fd2c4144', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:14,243 [238] [DEBUG] [app] Starting request: urn:request:634fceb6-331c-4aea-9e50-4735fd8d05dc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:54:14,243 [238] [DEBUG] [app] Ending request: urn:request:634fceb6-331c-4aea-9e50-4735fd8d05dc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:634fceb6-331c-4aea-9e50-4735fd8d05dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:54:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:54:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:54:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:14,254 [235] [DEBUG] [app] Ending request: urn:request:9bc654ae-431c-48a7-913b-eb7b8c77fa2c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9bc654ae-431c-48a7-913b-eb7b8c77fa2c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 03:54:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:54:14,255 [238] [DEBUG] [app] Ending request: urn:request:6c89fffd-8cc5-4654-9456-5e0f1c398d04 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c89fffd-8cc5-4654-9456-5e0f1c398d04', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:54:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
servicekey stdout | 2025-02-07 03:54:14,275 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:54:14,298 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:54:14,331 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:54:14,751 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:54:15,178 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:54:15,315 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:54:15,318 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:54:16,173 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:54:16,176 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:54:16,178 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:54:16,181 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:54:16,184 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:54:16,932 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:54:17,205 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:54:17,209 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:54:17,213 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:54:17,275 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:54:17,515 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:54:18,231 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:54:18,237 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:54:18,240 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:54:18,243 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:54:18,246 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:54:18,250 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:54:18,253 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:54:18,256 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:54:18,258 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:54:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:54:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:28.732147+00:00 (in 9.999498 seconds)
notificationworker stdout | 2025-02-07 03:54:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:28 UTC)" (scheduled at 2025-02-07 03:54:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:54:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:54:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 18, 732870), True, datetime.datetime(2025, 2, 7, 3, 54, 18, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:54:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:54:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:54:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:54:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:54:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:03.419110+00:00 (in 39.997738 seconds)
repositorygcworker stdout | 2025-02-07 03:54:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:23 UTC)" (scheduled at 2025-02-07 03:54:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:54:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:54:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:54:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:54:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:40.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 03:54:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:54 UTC)" (scheduled at 2025-02-07 03:54:24.423055+00:00)
gcworker stdout | 2025-02-07 03:54:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:54:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:54:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:54:26,498 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:54:28,437 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:54:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:54:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:38.732147+00:00 (in 9.999527 seconds)
notificationworker stdout | 2025-02-07 03:54:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:38 UTC)" (scheduled at 2025-02-07 03:54:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:54:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:54:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 28, 732840), True, datetime.datetime(2025, 2, 7, 3, 54, 28, 732840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:54:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:54:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:54:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:54:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:54:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:44.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 03:54:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:44 UTC)" (scheduled at 2025-02-07 03:54:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:54:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:54:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:54:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:54:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:54:29,235 [235] [DEBUG] [app] Starting request: urn:request:92815632-d706-4593-8a35-5ba5f77ef86d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:29,235 [238] [DEBUG] [app] Starting request: urn:request:394e2cd4-2d83-4ebe-aa19-42eeb26247d2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:54:29,238 [245] [DEBUG] [app] Starting request: urn:request:28ca0d8c-8361-487d-9d10-fa01f693ee26 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:29,238 [239] [DEBUG] [app] Starting request: urn:request:bcb84282-6d0e-40c0-91dc-cda7482b3231 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:29,239 [239] [DEBUG] [app] Ending request: urn:request:bcb84282-6d0e-40c0-91dc-cda7482b3231 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bcb84282-6d0e-40c0-91dc-cda7482b3231', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:54:29,239 [245] [DEBUG] [app] Ending request: urn:request:28ca0d8c-8361-487d-9d10-fa01f693ee26 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28ca0d8c-8361-487d-9d10-fa01f693ee26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:54:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:54:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:54:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:29,242 [235] [DEBUG] [app] Starting request: urn:request:e1d82147-fdec-42dc-8d3f-8cc5e440b758 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:29,242 [237] [DEBUG] [app] Starting request: urn:request:3ca70f92-aee7-40fb-9bd5-c4dda96ee12d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:29,242 [235] [DEBUG] [app] Ending request: urn:request:e1d82147-fdec-42dc-8d3f-8cc5e440b758 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e1d82147-fdec-42dc-8d3f-8cc5e440b758', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:29,242 [237] [DEBUG] [app] Ending request: urn:request:3ca70f92-aee7-40fb-9bd5-c4dda96ee12d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ca70f92-aee7-40fb-9bd5-c4dda96ee12d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:54:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:54:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:54:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:54:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:29,254 [235] [DEBUG] [app] Ending request: urn:request:92815632-d706-4593-8a35-5ba5f77ef86d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:92815632-d706-4593-8a35-5ba5f77ef86d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:54:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:54:29,255 [238] [DEBUG] [app] Ending request: urn:request:394e2cd4-2d83-4ebe-aa19-42eeb26247d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:394e2cd4-2d83-4ebe-aa19-42eeb26247d2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:54:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
queuecleanupworker stdout | 2025-02-07 03:54:29,507 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:54:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:54:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:32.625945+00:00 (in 1.004340 seconds)
namespacegcworker stdout | 2025-02-07 03:54:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:31 UTC)" (scheduled at 2025-02-07 03:54:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:54:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:54:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 31, 621879), True, datetime.datetime(2025, 2, 7, 3, 54, 31, 621879), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:54:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:54:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:54:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:54:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:54:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:31.621131+00:00 (in 58.994722 seconds)
namespacegcworker stdout | 2025-02-07 03:54:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:32 UTC)" (scheduled at 2025-02-07 03:54:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:54:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:54:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:54:33,831 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:54:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:54:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:03.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 03:54:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:03 UTC)" (scheduled at 2025-02-07 03:54:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:54:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 54, 33, 920002), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:54:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:54:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:54:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:54:34,810 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:56.713547+00:00 (in 18.997170 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:37 UTC)" (scheduled at 2025-02-07 03:54:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:54:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:54:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:48.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 03:54:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:48 UTC)" (scheduled at 2025-02-07 03:54:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:54:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:54:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 38, 732799), True, datetime.datetime(2025, 2, 7, 3, 54, 38, 732799), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:54:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:54:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:54:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:54:38,880 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:54:39,482 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:54:40,056 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:54:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:54:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:54.423055+00:00 (in 13.998344 seconds)
gcworker stdout | 2025-02-07 03:54:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:10 UTC)" (scheduled at 2025-02-07 03:54:40.424250+00:00)
gcworker stdout | 2025-02-07 03:54:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:54:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900180431, None, 1, 0])
gcworker stdout | 2025-02-07 03:54:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:54:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:54:41,202 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:54:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:54:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:11.831521+00:00 (in 29.999557 seconds)
autopruneworker stdout | 2025-02-07 03:54:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:11 UTC)" (scheduled at 2025-02-07 03:54:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:54:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896881836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:54:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:54:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:54:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:54:42,114 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:54:42,232 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:54:43,280 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:43,568 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:54:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:54:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:59.225378+00:00 (in 14.999556 seconds)
quotatotalworker stdout | 2025-02-07 03:54:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:59 UTC)" (scheduled at 2025-02-07 03:54:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:54:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:54:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:54:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:54:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:54:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:54:44,235 [235] [DEBUG] [app] Starting request: urn:request:5044809a-97d8-4f49-a61f-eee7faba607c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:44,235 [236] [DEBUG] [app] Starting request: urn:request:08145527-df46-48c8-83bc-fe6f7c66908b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:54:44,238 [246] [DEBUG] [app] Starting request: urn:request:82c59855-dde8-49f1-9fcc-b35a3067a63b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:44,238 [239] [DEBUG] [app] Starting request: urn:request:fc3f9bcc-c524-4da2-bc76-237be8c8b436 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:44,238 [239] [DEBUG] [app] Ending request: urn:request:fc3f9bcc-c524-4da2-bc76-237be8c8b436 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc3f9bcc-c524-4da2-bc76-237be8c8b436', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:54:44,238 [246] [DEBUG] [app] Ending request: urn:request:82c59855-dde8-49f1-9fcc-b35a3067a63b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82c59855-dde8-49f1-9fcc-b35a3067a63b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:54:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:54:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:54:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:44,241 [235] [DEBUG] [app] Starting request: urn:request:df6204bc-9e73-485f-b417-cd852463ce0e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:44,241 [235] [DEBUG] [app] Ending request: urn:request:df6204bc-9e73-485f-b417-cd852463ce0e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df6204bc-9e73-485f-b417-cd852463ce0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:44,242 [238] [DEBUG] [app] Starting request: urn:request:e9e84351-04fe-4164-b32d-9a6d3f30d320 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:54:44,242 [238] [DEBUG] [app] Ending request: urn:request:e9e84351-04fe-4164-b32d-9a6d3f30d320 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e9e84351-04fe-4164-b32d-9a6d3f30d320', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:54:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:54:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:54:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:44,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:44,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:44,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:44,253 [235] [DEBUG] [app] Ending request: urn:request:5044809a-97d8-4f49-a61f-eee7faba607c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5044809a-97d8-4f49-a61f-eee7faba607c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:54:44,253 [236] [DEBUG] [app] Ending request: urn:request:08145527-df46-48c8-83bc-fe6f7c66908b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08145527-df46-48c8-83bc-fe6f7c66908b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:54:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 03:54:44,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
servicekey stdout | 2025-02-07 03:54:44,311 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:54:44,334 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:54:44,352 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:54:44,765 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:54:45,214 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:54:45,344 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:54:45,353 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:54:46,183 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:54:46,186 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:54:46,190 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:54:46,193 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:54:46,197 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:54:46,968 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:54:47,214 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:54:47,217 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:54:47,219 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:54:47,311 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:54:47,523 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:54:48,243 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:54:48,249 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:54:48,252 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:54:48,255 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:54:48,259 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:54:48,262 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:54:48,265 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:54:48,268 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:54:48,271 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:54:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:54:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:50.814677+00:00 (in 2.082018 seconds)
notificationworker stdout | 2025-02-07 03:54:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:58 UTC)" (scheduled at 2025-02-07 03:54:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:54:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:54:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 48, 732986), True, datetime.datetime(2025, 2, 7, 3, 54, 48, 732986), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:54:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:54:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:54:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:54:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:54:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:54:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:58.732147+00:00 (in 7.916901 seconds)
notificationworker stdout | 2025-02-07 03:54:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:50 UTC)" (scheduled at 2025-02-07 03:54:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:54:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:54:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:54:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:54:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:54:59.937899+00:00 (in 6.999788 seconds)
exportactionlogsworker stdout | 2025-02-07 03:54:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:59:52 UTC)" (scheduled at 2025-02-07 03:54:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:54:52,939 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 54, 52, 938416), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:54:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 54, 52, 938416), True, datetime.datetime(2025, 2, 7, 3, 54, 52, 938416), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:54:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 54, 52, 938416), True, datetime.datetime(2025, 2, 7, 3, 54, 52, 938416), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 54, 52, 938416), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:54:52,949 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:54:52,949 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 03:59:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:54:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:54:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:10.424250+00:00 (in 16.000736 seconds)
gcworker stdout | 2025-02-07 03:54:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:24 UTC)" (scheduled at 2025-02-07 03:54:54.423055+00:00)
gcworker stdout | 2025-02-07 03:54:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:54:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:54:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:54:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:56.118855+00:00 (in 59.999438 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:54:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:56 UTC)" (scheduled at 2025-02-07 03:54:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:54:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:54:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:54:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:54:56,521 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:37.715878+00:00 (in 41.001872 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:56 UTC)" (scheduled at 2025-02-07 03:54:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:54:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:54:58,473 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:54:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:54:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:08.732147+00:00 (in 9.999478 seconds)
notificationworker stdout | 2025-02-07 03:54:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:08 UTC)" (scheduled at 2025-02-07 03:54:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:54:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:54:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 54, 58, 732973), True, datetime.datetime(2025, 2, 7, 3, 54, 58, 732973), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:54:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:54:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:54:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:54:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:54:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:14.225378+00:00 (in 14.999561 seconds)
quotatotalworker stdout | 2025-02-07 03:54:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:14 UTC)" (scheduled at 2025-02-07 03:54:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:54:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:54:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:54:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:54:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:54:59,236 [236] [DEBUG] [app] Starting request: urn:request:fb581bab-edfe-4fe4-a4d4-429714b6b604 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:59,236 [235] [DEBUG] [app] Starting request: urn:request:5c905eef-34bd-4947-9a41-9646812cbf44 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:54:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:54:59,239 [245] [DEBUG] [app] Starting request: urn:request:2af81b7e-0be9-4f69-9a95-3381d91f9d47 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:59,239 [246] [DEBUG] [app] Starting request: urn:request:633b1f68-2de3-478f-ade4-d5722a723c09 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:54:59,240 [245] [DEBUG] [app] Ending request: urn:request:2af81b7e-0be9-4f69-9a95-3381d91f9d47 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2af81b7e-0be9-4f69-9a95-3381d91f9d47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:54:59,240 [246] [DEBUG] [app] Ending request: urn:request:633b1f68-2de3-478f-ade4-d5722a723c09 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:633b1f68-2de3-478f-ade4-d5722a723c09', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:54:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:54:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:54:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:54:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:54:59,243 [238] [DEBUG] [app] Starting request: urn:request:bce0358f-7d47-4500-a971-dbef4271d009 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:59,243 [237] [DEBUG] [app] Starting request: urn:request:cc50f86a-95d2-435d-bcff-180640f0985d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:54:59,243 [238] [DEBUG] [app] Ending request: urn:request:bce0358f-7d47-4500-a971-dbef4271d009 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bce0358f-7d47-4500-a971-dbef4271d009', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:59,243 [237] [DEBUG] [app] Ending request: urn:request:cc50f86a-95d2-435d-bcff-180640f0985d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cc50f86a-95d2-435d-bcff-180640f0985d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:54:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:54:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:54:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:54:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:54:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:54:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:54:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:54:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:54:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:54:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:54:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:54:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:54:59,254 [235] [DEBUG] [app] Ending request: urn:request:5c905eef-34bd-4947-9a41-9646812cbf44 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5c905eef-34bd-4947-9a41-9646812cbf44', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:59,255 [236] [DEBUG] [app] Ending request: urn:request:fb581bab-edfe-4fe4-a4d4-429714b6b604 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb581bab-edfe-4fe4-a4d4-429714b6b604', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:54:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:54:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:54:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:54:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:54:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:54:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 03:54:59,542 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:54:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:54:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:04.936688+00:00 (in 4.998328 seconds)
exportactionlogsworker stdout | 2025-02-07 03:54:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:59 UTC)" (scheduled at 2025-02-07 03:54:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:54:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:54:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:55:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:55:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:55:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:23.420886+00:00 (in 20.001298 seconds)
repositorygcworker stdout | 2025-02-07 03:55:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:03 UTC)" (scheduled at 2025-02-07 03:55:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:55:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:55:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 3, 419794), True, datetime.datetime(2025, 2, 7, 3, 55, 3, 419794), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:55:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:55:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:55:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:55:03,843 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:55:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:55:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:33.919286+00:00 (in 29.999551 seconds)
buildlogsarchiver stdout | 2025-02-07 03:55:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:33 UTC)" (scheduled at 2025-02-07 03:55:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:55:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 55, 3, 919986), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:55:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:55:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:55:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:55:04,846 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:55:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:55:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:59.937899+00:00 (in 55.000766 seconds)
exportactionlogsworker stdout | 2025-02-07 03:55:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:04 UTC)" (scheduled at 2025-02-07 03:55:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:55:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:55:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 4, 937491), True, datetime.datetime(2025, 2, 7, 3, 55, 4, 937491), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:55:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:55:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:55:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:55:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:55:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:18.732147+00:00 (in 9.999491 seconds)
notificationworker stdout | 2025-02-07 03:55:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:18 UTC)" (scheduled at 2025-02-07 03:55:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:55:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:55:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 8, 732969), True, datetime.datetime(2025, 2, 7, 3, 55, 8, 732969), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:55:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:55:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:55:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:55:08,916 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:55:09,518 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:55:10,093 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:55:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:55:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:24.423055+00:00 (in 13.998350 seconds)
gcworker stdout | 2025-02-07 03:55:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:40 UTC)" (scheduled at 2025-02-07 03:55:10.424250+00:00)
gcworker stdout | 2025-02-07 03:55:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:55:10,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900210431, None, 1, 0])
gcworker stdout | 2025-02-07 03:55:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:55:10,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:55:11,234 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:55:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:55:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:41.831521+00:00 (in 29.999525 seconds)
autopruneworker stdout | 2025-02-07 03:55:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:41 UTC)" (scheduled at 2025-02-07 03:55:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:55:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896911836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:55:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:55:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:55:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:55:12,141 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:55:12,268 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:55:13,317 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:13,605 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:55:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:55:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:29.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 03:55:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:29 UTC)" (scheduled at 2025-02-07 03:55:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:55:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:55:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:55:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:55:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:55:14,235 [235] [DEBUG] [app] Starting request: urn:request:612e7ba6-1c5c-4bfe-a2de-f825e4889e4e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:14,235 [237] [DEBUG] [app] Starting request: urn:request:77eee031-4985-46eb-ada9-66b911bdf060 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:55:14,238 [246] [DEBUG] [app] Starting request: urn:request:1c1b07c8-2ccb-4f18-856f-7a63508146cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:14,238 [245] [DEBUG] [app] Starting request: urn:request:76857aeb-184e-4570-b38e-9024f8c52b61 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:14,239 [246] [DEBUG] [app] Ending request: urn:request:1c1b07c8-2ccb-4f18-856f-7a63508146cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c1b07c8-2ccb-4f18-856f-7a63508146cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:55:14,239 [245] [DEBUG] [app] Ending request: urn:request:76857aeb-184e-4570-b38e-9024f8c52b61 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76857aeb-184e-4570-b38e-9024f8c52b61', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:55:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:55:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:55:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:14,242 [237] [DEBUG] [app] Starting request: urn:request:340503c0-7a21-4a8b-86f7-8661be5bc099 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:55:14,242 [237] [DEBUG] [app] Ending request: urn:request:340503c0-7a21-4a8b-86f7-8661be5bc099 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:340503c0-7a21-4a8b-86f7-8661be5bc099', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:55:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:14,242 [238] [DEBUG] [app] Starting request: urn:request:2ada16c4-1a85-4079-9f1e-a88fddfff8f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:14,243 [238] [DEBUG] [app] Ending request: urn:request:2ada16c4-1a85-4079-9f1e-a88fddfff8f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2ada16c4-1a85-4079-9f1e-a88fddfff8f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:55:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:55:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:14,254 [237] [DEBUG] [app] Ending request: urn:request:77eee031-4985-46eb-ada9-66b911bdf060 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77eee031-4985-46eb-ada9-66b911bdf060', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:55:14,254 [235] [DEBUG] [app] Ending request: urn:request:612e7ba6-1c5c-4bfe-a2de-f825e4889e4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:612e7ba6-1c5c-4bfe-a2de-f825e4889e4e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:55:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:55:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
servicekey stdout | 2025-02-07 03:55:14,342 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:55:14,371 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:55:14,388 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:55:14,798 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:55:15,251 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:55:15,380 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:55:15,384 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:55:16,191 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:55:16,194 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:55:16,197 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:55:16,201 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:55:16,204 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:55:17,005 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:55:17,224 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:55:17,227 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:55:17,230 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:55:17,348 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:55:17,532 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:55:18,255 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:55:18,259 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:55:18,262 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:55:18,265 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:55:18,268 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:55:18,271 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:55:18,275 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:55:18,281 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:55:18,284 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:55:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:55:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:28.732147+00:00 (in 9.999508 seconds)
notificationworker stdout | 2025-02-07 03:55:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:28 UTC)" (scheduled at 2025-02-07 03:55:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:55:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:55:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 18, 732850), True, datetime.datetime(2025, 2, 7, 3, 55, 18, 732850), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:55:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:55:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:55:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:55:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:55:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:03.419110+00:00 (in 39.997690 seconds)
repositorygcworker stdout | 2025-02-07 03:55:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:23 UTC)" (scheduled at 2025-02-07 03:55:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:55:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:55:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:55:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:55:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:40.424250+00:00 (in 16.000761 seconds)
gcworker stdout | 2025-02-07 03:55:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:54 UTC)" (scheduled at 2025-02-07 03:55:24.423055+00:00)
gcworker stdout | 2025-02-07 03:55:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:55:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:55:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:55:26,534 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:55:28,484 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:55:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:55:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:38.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 03:55:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:38 UTC)" (scheduled at 2025-02-07 03:55:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:55:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:55:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 28, 732883), True, datetime.datetime(2025, 2, 7, 3, 55, 28, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:55:28,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:55:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:55:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:55:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:55:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:44.225378+00:00 (in 14.999525 seconds)
quotatotalworker stdout | 2025-02-07 03:55:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:44 UTC)" (scheduled at 2025-02-07 03:55:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:55:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:55:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:55:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:55:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:55:29,235 [238] [DEBUG] [app] Starting request: urn:request:f4ec364e-cb78-49db-843f-cb361e077539 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:29,235 [235] [DEBUG] [app] Starting request: urn:request:c8164aba-0716-4103-aaee-f836b88973d4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:55:29,238 [239] [DEBUG] [app] Starting request: urn:request:4d59832d-8e84-483e-adcf-b648624b35df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:29,238 [246] [DEBUG] [app] Starting request: urn:request:a1dd5724-2222-43b7-8824-f95c2c2cf8bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:29,238 [239] [DEBUG] [app] Ending request: urn:request:4d59832d-8e84-483e-adcf-b648624b35df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d59832d-8e84-483e-adcf-b648624b35df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:55:29,238 [246] [DEBUG] [app] Ending request: urn:request:a1dd5724-2222-43b7-8824-f95c2c2cf8bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a1dd5724-2222-43b7-8824-f95c2c2cf8bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:55:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:55:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:29,241 [238] [DEBUG] [app] Starting request: urn:request:ec47e222-ddb5-40ab-b693-1fa2d90805bb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:55:29,242 [238] [DEBUG] [app] Ending request: urn:request:ec47e222-ddb5-40ab-b693-1fa2d90805bb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec47e222-ddb5-40ab-b693-1fa2d90805bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:55:29,242 [235] [DEBUG] [app] Starting request: urn:request:f6c73a53-e822-4256-bdee-5bd0a313933a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:29,242 [235] [DEBUG] [app] Ending request: urn:request:f6c73a53-e822-4256-bdee-5bd0a313933a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6c73a53-e822-4256-bdee-5bd0a313933a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:55:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:55:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:55:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:29,253 [238] [DEBUG] [app] Ending request: urn:request:f4ec364e-cb78-49db-843f-cb361e077539 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f4ec364e-cb78-49db-843f-cb361e077539', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:29,253 [235] [DEBUG] [app] Ending request: urn:request:c8164aba-0716-4103-aaee-f836b88973d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c8164aba-0716-4103-aaee-f836b88973d4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:55:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:55:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:55:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 03:55:29,579 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:55:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:55:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:32.625945+00:00 (in 1.004345 seconds)
namespacegcworker stdout | 2025-02-07 03:55:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:31 UTC)" (scheduled at 2025-02-07 03:55:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:55:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:55:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 31, 621924), True, datetime.datetime(2025, 2, 7, 3, 55, 31, 621924), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:55:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:55:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:55:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:55:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:55:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:31.621131+00:00 (in 58.994710 seconds)
namespacegcworker stdout | 2025-02-07 03:55:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:32 UTC)" (scheduled at 2025-02-07 03:55:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:55:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:55:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:32 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:55:33,878 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:55:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:55:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:03.919286+00:00 (in 29.999552 seconds)
buildlogsarchiver stdout | 2025-02-07 03:55:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:03 UTC)" (scheduled at 2025-02-07 03:55:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:55:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 55, 33, 920026), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:55:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:55:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:55:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:03 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:55:34,858 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:56.713547+00:00 (in 18.997198 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:37 UTC)" (scheduled at 2025-02-07 03:55:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:55:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:55:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:48.732147+00:00 (in 9.999540 seconds)
notificationworker stdout | 2025-02-07 03:55:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:48 UTC)" (scheduled at 2025-02-07 03:55:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:55:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:55:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 38, 732892), True, datetime.datetime(2025, 2, 7, 3, 55, 38, 732892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:55:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:55:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:55:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:55:38,953 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:55:39,526 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:55:40,118 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:55:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:55:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:54.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 03:55:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:10 UTC)" (scheduled at 2025-02-07 03:55:40.424250+00:00)
gcworker stdout | 2025-02-07 03:55:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:55:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900240430, None, 1, 0])
gcworker stdout | 2025-02-07 03:55:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:55:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:55:41,270 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:55:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:55:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:11.831521+00:00 (in 29.999480 seconds)
autopruneworker stdout | 2025-02-07 03:55:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:11 UTC)" (scheduled at 2025-02-07 03:55:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:55:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896941836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:55:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:55:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:55:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:55:42,177 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:55:42,281 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:55:43,337 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:43,641 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:55:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:55:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:59.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 03:55:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:59 UTC)" (scheduled at 2025-02-07 03:55:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:55:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:55:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:55:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:55:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:55:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:55:44,235 [237] [DEBUG] [app] Starting request: urn:request:d7d3cb7a-6358-40ac-b110-69f95ee5bcff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:44,235 [235] [DEBUG] [app] Starting request: urn:request:9a475c8e-fc5a-4f97-b6e7-c5089baba0ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:55:44,238 [246] [DEBUG] [app] Starting request: urn:request:11b09378-f648-492f-add5-2d20095f7f32 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:44,238 [243] [DEBUG] [app] Starting request: urn:request:502b4a31-ce47-4801-83d6-ad36635116ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:44,239 [246] [DEBUG] [app] Ending request: urn:request:11b09378-f648-492f-add5-2d20095f7f32 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11b09378-f648-492f-add5-2d20095f7f32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:55:44,239 [243] [DEBUG] [app] Ending request: urn:request:502b4a31-ce47-4801-83d6-ad36635116ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:502b4a31-ce47-4801-83d6-ad36635116ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:55:44,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:55:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 03:55:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:44,241 [235] [DEBUG] [app] Starting request: urn:request:fac9b2ac-dea7-41cf-bb9e-e98ef0b4f048 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:55:44,242 [235] [DEBUG] [app] Ending request: urn:request:fac9b2ac-dea7-41cf-bb9e-e98ef0b4f048 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fac9b2ac-dea7-41cf-bb9e-e98ef0b4f048', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:55:44,242 [238] [DEBUG] [app] Starting request: urn:request:f3c73f9c-eee0-4e48-b769-a1f1e38b43fe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:55:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:44,242 [238] [DEBUG] [app] Ending request: urn:request:f3c73f9c-eee0-4e48-b769-a1f1e38b43fe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3c73f9c-eee0-4e48-b769-a1f1e38b43fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:55:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:55:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:44,253 [235] [DEBUG] [app] Ending request: urn:request:9a475c8e-fc5a-4f97-b6e7-c5089baba0ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a475c8e-fc5a-4f97-b6e7-c5089baba0ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:44,253 [237] [DEBUG] [app] Ending request: urn:request:d7d3cb7a-6358-40ac-b110-69f95ee5bcff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d7d3cb7a-6358-40ac-b110-69f95ee5bcff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:55:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:55:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:55:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
servicekey stdout | 2025-02-07 03:55:44,378 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:55:44,394 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:55:44,425 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:55:44,829 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:55:45,287 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:55:45,410 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:55:45,419 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:55:46,200 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:55:46,204 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:55:46,206 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:55:46,210 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:55:46,212 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:55:47,023 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:55:47,231 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:55:47,235 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:55:47,237 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:55:47,385 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:55:47,562 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:55:48,267 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:55:48,271 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:55:48,273 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:55:48,277 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:55:48,280 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:55:48,283 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:55:48,285 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:55:48,291 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:55:48,294 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:55:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:55:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:50.814677+00:00 (in 2.082088 seconds)
notificationworker stdout | 2025-02-07 03:55:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:58 UTC)" (scheduled at 2025-02-07 03:55:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:55:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:55:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 48, 732908), True, datetime.datetime(2025, 2, 7, 3, 55, 48, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:55:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:55:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:55:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:55:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:55:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:55:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:55:58.732147+00:00 (in 7.917007 seconds)
notificationworker stdout | 2025-02-07 03:55:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:50 UTC)" (scheduled at 2025-02-07 03:55:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:55:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:55:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:55:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:55:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 03:55:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:24 UTC)" (scheduled at 2025-02-07 03:55:54.423055+00:00)
gcworker stdout | 2025-02-07 03:55:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:55:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:55:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:55:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:56.118855+00:00 (in 59.999467 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:55:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:56 UTC)" (scheduled at 2025-02-07 03:55:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:55:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:55:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:55:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:55:56,570 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:37.715878+00:00 (in 41.001839 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:56 UTC)" (scheduled at 2025-02-07 03:55:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:55:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:55:58,520 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:55:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:55:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:08.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 03:55:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:08 UTC)" (scheduled at 2025-02-07 03:55:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:55:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:55:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 55, 58, 732855), True, datetime.datetime(2025, 2, 7, 3, 55, 58, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:55:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:55:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:55:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:55:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:55:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:14.225378+00:00 (in 14.999569 seconds)
quotatotalworker stdout | 2025-02-07 03:55:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:14 UTC)" (scheduled at 2025-02-07 03:55:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:55:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:55:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:55:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:55:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:55:59,236 [235] [DEBUG] [app] Starting request: urn:request:3d0769c2-8bad-47bb-be19-4d80abb9b16f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:59,236 [237] [DEBUG] [app] Starting request: urn:request:4ba5278a-ae06-42f1-9fb7-677eb1cfd15d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:55:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:55:59,238 [239] [DEBUG] [app] Starting request: urn:request:8057b467-0c4a-475a-a0f3-84a62778867c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:59,238 [246] [DEBUG] [app] Starting request: urn:request:4cbe4874-8302-4f2f-b62d-b2f0f216801d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:55:59,239 [239] [DEBUG] [app] Ending request: urn:request:8057b467-0c4a-475a-a0f3-84a62778867c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8057b467-0c4a-475a-a0f3-84a62778867c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:55:59,239 [246] [DEBUG] [app] Ending request: urn:request:4cbe4874-8302-4f2f-b62d-b2f0f216801d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4cbe4874-8302-4f2f-b62d-b2f0f216801d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:55:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:55:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:55:59,241 [235] [DEBUG] [app] Starting request: urn:request:ba442e7a-e022-4cde-97ce-25f81ae92267 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:55:59,242 [235] [DEBUG] [app] Ending request: urn:request:ba442e7a-e022-4cde-97ce-25f81ae92267 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba442e7a-e022-4cde-97ce-25f81ae92267', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:55:59,242 [238] [DEBUG] [app] Starting request: urn:request:38f9e2c7-1d7e-4780-855e-8c048a2393dc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:59,242 [238] [DEBUG] [app] Ending request: urn:request:38f9e2c7-1d7e-4780-855e-8c048a2393dc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:38f9e2c7-1d7e-4780-855e-8c048a2393dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:55:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:55:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:55:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:55:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:55:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:55:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:55:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:55:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:55:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:55:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:55:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:55:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:55:59,253 [235] [DEBUG] [app] Ending request: urn:request:3d0769c2-8bad-47bb-be19-4d80abb9b16f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d0769c2-8bad-47bb-be19-4d80abb9b16f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:59,253 [237] [DEBUG] [app] Ending request: urn:request:4ba5278a-ae06-42f1-9fb7-677eb1cfd15d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ba5278a-ae06-42f1-9fb7-677eb1cfd15d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:55:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:55:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:55:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:55:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 03:55:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:55:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 03:55:59,593 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:55:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:55:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:04.936688+00:00 (in 4.998292 seconds)
exportactionlogsworker stdout | 2025-02-07 03:55:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:59 UTC)" (scheduled at 2025-02-07 03:55:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:55:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:55:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:56:59 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:56:00,816 [60] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2025-02-07 03:56:00,816 [60] [INFO] [apscheduler.executors.default] Running job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2025-02-07 03:56:00 UTC)" (scheduled at 2025-02-07 03:56:00.815400+00:00)
expiredappspecifictokenworker stdout | 2025-02-07 03:56:00,816 [60] [DEBUG] [__main__] Garbage collecting expired app specific tokens with window: 1 day, 0:00:00
expiredappspecifictokenworker stdout | 2025-02-07 03:56:00,817 [60] [DEBUG] [peewee] ('DELETE FROM "appspecificauthtoken" WHERE ("appspecificauthtoken"."expiration" < %s)', [datetime.datetime(2025, 2, 6, 3, 56, 0, 817106)])
expiredappspecifictokenworker stdout | 2025-02-07 03:56:00,817 [60] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:00.815400+00:00 (in 3599.997681 seconds)
expiredappspecifictokenworker stdout | 2025-02-07 03:56:00,823 [60] [DEBUG] [data.database] Disconnecting from database.
expiredappspecifictokenworker stdout | 2025-02-07 03:56:00,823 [60] [INFO] [apscheduler.executors.default] Job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2025-02-07 04:56:00 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:56:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:56:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:23.420886+00:00 (in 20.001301 seconds)
repositorygcworker stdout | 2025-02-07 03:56:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:03 UTC)" (scheduled at 2025-02-07 03:56:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:56:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:56:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 3, 419789), True, datetime.datetime(2025, 2, 7, 3, 56, 3, 419789), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:56:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:56:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:56:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:56:03,910 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
buildlogsarchiver stdout | 2025-02-07 03:56:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:56:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:33.919286+00:00 (in 29.999555 seconds)
buildlogsarchiver stdout | 2025-02-07 03:56:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:33 UTC)" (scheduled at 2025-02-07 03:56:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:56:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 56, 3, 919984), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:56:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:56:03,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:56:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:33 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:56:04,895 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:56:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:56:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:59.937899+00:00 (in 55.000761 seconds)
exportactionlogsworker stdout | 2025-02-07 03:56:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:04 UTC)" (scheduled at 2025-02-07 03:56:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:56:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:56:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 4, 937449), True, datetime.datetime(2025, 2, 7, 3, 56, 4, 937449), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:56:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:56:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:56:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:56:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:56:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:18.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 03:56:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:18 UTC)" (scheduled at 2025-02-07 03:56:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:56:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:56:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 8, 732885), True, datetime.datetime(2025, 2, 7, 3, 56, 8, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:56:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:56:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:56:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:56:08,969 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:56:09,534 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:56:10,137 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:56:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:56:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:24.423055+00:00 (in 13.998323 seconds)
gcworker stdout | 2025-02-07 03:56:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:40 UTC)" (scheduled at 2025-02-07 03:56:10.424250+00:00)
gcworker stdout | 2025-02-07 03:56:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:56:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900270431, None, 1, 0])
gcworker stdout | 2025-02-07 03:56:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:56:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:56:11,307 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:56:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:56:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:41.831521+00:00 (in 29.999514 seconds)
autopruneworker stdout | 2025-02-07 03:56:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:41 UTC)" (scheduled at 2025-02-07 03:56:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:56:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738896971836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:56:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:56:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:56:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:56:12,213 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:56:12,301 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:56:13,373 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:13,674 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:56:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:56:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:29.225378+00:00 (in 14.999515 seconds)
quotatotalworker stdout | 2025-02-07 03:56:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:29 UTC)" (scheduled at 2025-02-07 03:56:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:56:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:56:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:56:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:56:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:56:14,235 [235] [DEBUG] [app] Starting request: urn:request:47f0bb55-dced-4670-99cf-0c5f4e50f4f0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:14,235 [236] [DEBUG] [app] Starting request: urn:request:37b64c72-52f1-4231-9bd3-249dcfd59ec9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:56:14,238 [246] [DEBUG] [app] Starting request: urn:request:2ea0005f-5946-4f60-9d02-b31ba3ca098a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:14,238 [245] [DEBUG] [app] Starting request: urn:request:c780303d-adf3-4b57-96ad-d2348a75ad6a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:14,238 [246] [DEBUG] [app] Ending request: urn:request:2ea0005f-5946-4f60-9d02-b31ba3ca098a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ea0005f-5946-4f60-9d02-b31ba3ca098a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:56:14,239 [245] [DEBUG] [app] Ending request: urn:request:c780303d-adf3-4b57-96ad-d2348a75ad6a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c780303d-adf3-4b57-96ad-d2348a75ad6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:56:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:56:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:14,242 [235] [DEBUG] [app] Starting request: urn:request:d13e1bc4-6438-4cd9-ac5e-f25fa296fb62 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:14,242 [235] [DEBUG] [app] Ending request: urn:request:d13e1bc4-6438-4cd9-ac5e-f25fa296fb62 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d13e1bc4-6438-4cd9-ac5e-f25fa296fb62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:14,242 [236] [DEBUG] [app] Starting request: urn:request:50f69d45-db12-4064-8408-53ec94f1c920 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:14,242 [236] [DEBUG] [app] Ending request: urn:request:50f69d45-db12-4064-8408-53ec94f1c920 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50f69d45-db12-4064-8408-53ec94f1c920', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:56:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:14,254 [236] [DEBUG] [app] Ending request: urn:request:37b64c72-52f1-4231-9bd3-249dcfd59ec9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37b64c72-52f1-4231-9bd3-249dcfd59ec9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:56:14,255 [235] [DEBUG] [app] Ending request: urn:request:47f0bb55-dced-4670-99cf-0c5f4e50f4f0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47f0bb55-dced-4670-99cf-0c5f4e50f4f0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:56:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:56:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
servicekey stdout | 2025-02-07 03:56:14,406 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:56:14,426 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:56:14,461 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:56:14,857 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:56:15,314 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:56:15,447 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:56:15,456 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:56:16,209 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:56:16,213 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:56:16,217 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:56:16,222 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:56:16,225 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:56:17,059 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:56:17,240 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:56:17,242 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:56:17,245 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:56:17,421 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:56:17,573 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:56:18,277 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:56:18,281 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:56:18,284 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:56:18,288 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:56:18,291 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:56:18,293 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:56:18,296 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:56:18,300 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:56:18,303 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:56:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:56:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:28.732147+00:00 (in 9.999533 seconds)
notificationworker stdout | 2025-02-07 03:56:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:28 UTC)" (scheduled at 2025-02-07 03:56:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:56:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:56:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 18, 732908), True, datetime.datetime(2025, 2, 7, 3, 56, 18, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:56:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:56:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:56:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:56:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:56:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:03.419110+00:00 (in 39.997729 seconds)
repositorygcworker stdout | 2025-02-07 03:56:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:23 UTC)" (scheduled at 2025-02-07 03:56:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:56:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:56:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:56:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:56:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:40.424250+00:00 (in 16.000744 seconds)
gcworker stdout | 2025-02-07 03:56:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:54 UTC)" (scheduled at 2025-02-07 03:56:24.423055+00:00)
gcworker stdout | 2025-02-07 03:56:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:56:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:56:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:56:26,588 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:56:28,537 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:56:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:56:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:38.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:56:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:38 UTC)" (scheduled at 2025-02-07 03:56:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:56:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:56:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 28, 732868), True, datetime.datetime(2025, 2, 7, 3, 56, 28, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:56:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:56:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:56:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:56:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:56:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:44.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 03:56:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:44 UTC)" (scheduled at 2025-02-07 03:56:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:56:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:56:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:56:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:56:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:56:29,236 [235] [DEBUG] [app] Starting request: urn:request:4572eb2b-0f92-46a1-ae0f-9437be4f62c5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:29,236 [238] [DEBUG] [app] Starting request: urn:request:c697cd71-0458-4d4e-bd8d-b31a50ae1019 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:56:29,240 [246] [DEBUG] [app] Starting request: urn:request:3272219f-0348-4b78-92e5-ef08bd9542fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:29,240 [245] [DEBUG] [app] Starting request: urn:request:de072f52-9f8a-4e8c-ac19-bdd1976812cf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:29,240 [246] [DEBUG] [app] Ending request: urn:request:3272219f-0348-4b78-92e5-ef08bd9542fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3272219f-0348-4b78-92e5-ef08bd9542fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:56:29,240 [245] [DEBUG] [app] Ending request: urn:request:de072f52-9f8a-4e8c-ac19-bdd1976812cf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:de072f52-9f8a-4e8c-ac19-bdd1976812cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:56:29,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:56:29,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 03:56:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:29,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:29,244 [236] [DEBUG] [app] Starting request: urn:request:659fad3c-8f45-48d0-9bb7-6d8c4ce7b9ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:29,244 [237] [DEBUG] [app] Starting request: urn:request:900db877-f1f6-488f-b97c-fc755d01118d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:29,244 [236] [DEBUG] [app] Ending request: urn:request:659fad3c-8f45-48d0-9bb7-6d8c4ce7b9ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:659fad3c-8f45-48d0-9bb7-6d8c4ce7b9ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:29,244 [237] [DEBUG] [app] Ending request: urn:request:900db877-f1f6-488f-b97c-fc755d01118d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:900db877-f1f6-488f-b97c-fc755d01118d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:29,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:56:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:29,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:29,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:29,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:29,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:29,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:29,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:29,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:29,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:29,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:29,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:29,256 [238] [DEBUG] [app] Ending request: urn:request:c697cd71-0458-4d4e-bd8d-b31a50ae1019 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c697cd71-0458-4d4e-bd8d-b31a50ae1019', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:29,256 [235] [DEBUG] [app] Ending request: urn:request:4572eb2b-0f92-46a1-ae0f-9437be4f62c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4572eb2b-0f92-46a1-ae0f-9437be4f62c5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:29,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:56:29,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:56:29,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:56:29,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
queuecleanupworker stdout | 2025-02-07 03:56:29,626 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:56:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:56:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:32.625945+00:00 (in 1.004367 seconds)
namespacegcworker stdout | 2025-02-07 03:56:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:31 UTC)" (scheduled at 2025-02-07 03:56:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:56:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:56:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 31, 621849), True, datetime.datetime(2025, 2, 7, 3, 56, 31, 621849), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:56:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:56:31,628 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:56:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:56:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:56:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:13.624301+00:00 (in 40.997899 seconds)
namespacegcworker stdout | 2025-02-07 03:56:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:32 UTC)" (scheduled at 2025-02-07 03:56:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:56:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:56:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:56:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:56:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:03.919286+00:00 (in 29.999564 seconds)
buildlogsarchiver stdout | 2025-02-07 03:56:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:03 UTC)" (scheduled at 2025-02-07 03:56:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:56:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 56, 33, 919985), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:56:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:56:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:56:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:56:33,941 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:56:34,917 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:56.713547+00:00 (in 18.997169 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:37 UTC)" (scheduled at 2025-02-07 03:56:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:56:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:56:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:48.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 03:56:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:48 UTC)" (scheduled at 2025-02-07 03:56:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:56:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:56:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 38, 732914), True, datetime.datetime(2025, 2, 7, 3, 56, 38, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:56:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:56:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:56:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:56:38,991 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:56:39,570 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:56:40,174 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:56:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:56:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:54.423055+00:00 (in 13.998357 seconds)
gcworker stdout | 2025-02-07 03:56:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:10 UTC)" (scheduled at 2025-02-07 03:56:40.424250+00:00)
gcworker stdout | 2025-02-07 03:56:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:56:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900300431, None, 1, 0])
gcworker stdout | 2025-02-07 03:56:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:56:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:56:41,343 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:56:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:56:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:11.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 03:56:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:11 UTC)" (scheduled at 2025-02-07 03:56:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:56:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897001836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:56:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:56:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:56:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:56:42,247 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:56:42,318 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:56:43,388 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:43,711 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:56:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:56:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:59.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 03:56:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:59 UTC)" (scheduled at 2025-02-07 03:56:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:56:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:56:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:56:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:56:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:56:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:56:44,235 [235] [DEBUG] [app] Starting request: urn:request:cc0a4aa0-d076-4e78-8ce0-4406e291e083 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:44,235 [238] [DEBUG] [app] Starting request: urn:request:f76e0001-5046-4564-9533-352662aa711a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:56:44,238 [245] [DEBUG] [app] Starting request: urn:request:46532d62-8896-4edf-a1d8-f07f0ed5b2db (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:44,238 [239] [DEBUG] [app] Starting request: urn:request:248d6802-bf46-4cd4-a3b7-08195158078b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:44,239 [245] [DEBUG] [app] Ending request: urn:request:46532d62-8896-4edf-a1d8-f07f0ed5b2db (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:46532d62-8896-4edf-a1d8-f07f0ed5b2db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:56:44,239 [239] [DEBUG] [app] Ending request: urn:request:248d6802-bf46-4cd4-a3b7-08195158078b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:248d6802-bf46-4cd4-a3b7-08195158078b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:56:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:56:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:44,243 [237] [DEBUG] [app] Starting request: urn:request:986ed1a5-bea8-4897-8e24-d798ea489a37 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:44,243 [236] [DEBUG] [app] Starting request: urn:request:d9fde693-887e-4737-8b3c-7d32a1900325 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:44,243 [236] [DEBUG] [app] Ending request: urn:request:d9fde693-887e-4737-8b3c-7d32a1900325 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9fde693-887e-4737-8b3c-7d32a1900325', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:44,243 [237] [DEBUG] [app] Ending request: urn:request:986ed1a5-bea8-4897-8e24-d798ea489a37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:986ed1a5-bea8-4897-8e24-d798ea489a37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:56:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:44,256 [238] [DEBUG] [app] Ending request: urn:request:f76e0001-5046-4564-9533-352662aa711a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f76e0001-5046-4564-9533-352662aa711a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:44,256 [235] [DEBUG] [app] Ending request: urn:request:cc0a4aa0-d076-4e78-8ce0-4406e291e083 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cc0a4aa0-d076-4e78-8ce0-4406e291e083', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:56:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 03:56:44,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:56:44,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
servicekey stdout | 2025-02-07 03:56:44,442 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:56:44,461 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:56:44,497 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:56:44,889 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:56:45,329 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:56:45,467 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:56:45,472 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:56:46,217 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:56:46,220 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:56:46,223 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:56:46,228 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:56:46,233 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:56:47,078 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:56:47,248 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:56:47,251 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:56:47,254 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:56:47,436 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:56:47,582 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:56:48,287 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:56:48,290 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:56:48,293 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:56:48,297 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:56:48,300 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:56:48,302 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:56:48,307 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:56:48,311 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:56:48,315 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:56:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:56:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:50.814677+00:00 (in 2.082046 seconds)
notificationworker stdout | 2025-02-07 03:56:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:58 UTC)" (scheduled at 2025-02-07 03:56:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:56:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:56:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 48, 732972), True, datetime.datetime(2025, 2, 7, 3, 56, 48, 732972), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:56:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:56:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:56:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:56:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:56:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:56:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:56:58.732147+00:00 (in 7.917022 seconds)
notificationworker stdout | 2025-02-07 03:56:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:50 UTC)" (scheduled at 2025-02-07 03:56:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:56:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:56:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:56:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:56:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:10.424250+00:00 (in 16.000731 seconds)
gcworker stdout | 2025-02-07 03:56:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:24 UTC)" (scheduled at 2025-02-07 03:56:54.423055+00:00)
gcworker stdout | 2025-02-07 03:56:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:56:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:56:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:56:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:56.118855+00:00 (in 59.999507 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:56:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:56 UTC)" (scheduled at 2025-02-07 03:56:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:56:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:56:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:56:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:56:56,624 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:37.715878+00:00 (in 41.001898 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:56 UTC)" (scheduled at 2025-02-07 03:56:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:56:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:56:58,573 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:56:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:56:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:08.732147+00:00 (in 9.999539 seconds)
notificationworker stdout | 2025-02-07 03:56:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:08 UTC)" (scheduled at 2025-02-07 03:56:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:56:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:56:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 56, 58, 732893), True, datetime.datetime(2025, 2, 7, 3, 56, 58, 732893), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:56:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:56:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:56:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:56:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:56:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:14.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 03:56:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:14 UTC)" (scheduled at 2025-02-07 03:56:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:56:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:56:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:56:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:56:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:56:59,236 [237] [DEBUG] [app] Starting request: urn:request:249008fb-a597-4e7c-85b7-72072d7eed22 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:59,236 [235] [DEBUG] [app] Starting request: urn:request:e07e78bd-a3ac-45e5-81b6-8da0271df8a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:56:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:56:59,239 [239] [DEBUG] [app] Starting request: urn:request:1ed7e28b-1de4-4c58-8e5e-0f514aabd919 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:59,239 [246] [DEBUG] [app] Starting request: urn:request:92d68f54-02e4-4459-bad1-c01a6a59e3b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:56:59,239 [246] [DEBUG] [app] Ending request: urn:request:92d68f54-02e4-4459-bad1-c01a6a59e3b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:92d68f54-02e4-4459-bad1-c01a6a59e3b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:56:59,239 [239] [DEBUG] [app] Ending request: urn:request:1ed7e28b-1de4-4c58-8e5e-0f514aabd919 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1ed7e28b-1de4-4c58-8e5e-0f514aabd919', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:56:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:56:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 03:56:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:56:59,242 [238] [DEBUG] [app] Starting request: urn:request:055d295a-47d5-41b8-b737-08b8ebefcd04 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:59,242 [237] [DEBUG] [app] Starting request: urn:request:4025a70f-bb38-4302-906a-2001620faec8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:56:59,242 [238] [DEBUG] [app] Ending request: urn:request:055d295a-47d5-41b8-b737-08b8ebefcd04 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:055d295a-47d5-41b8-b737-08b8ebefcd04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:59,242 [237] [DEBUG] [app] Ending request: urn:request:4025a70f-bb38-4302-906a-2001620faec8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4025a70f-bb38-4302-906a-2001620faec8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:56:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:56:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:56:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:56:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:56:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:56:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:56:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:56:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:56:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:56:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:56:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:56:59,254 [235] [DEBUG] [app] Ending request: urn:request:e07e78bd-a3ac-45e5-81b6-8da0271df8a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e07e78bd-a3ac-45e5-81b6-8da0271df8a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:59,255 [237] [DEBUG] [app] Ending request: urn:request:249008fb-a597-4e7c-85b7-72072d7eed22 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:249008fb-a597-4e7c-85b7-72072d7eed22', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:56:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:56:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:56:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:56:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:56:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:56:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 03:56:59,662 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:56:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:56:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:04.936688+00:00 (in 4.998260 seconds)
exportactionlogsworker stdout | 2025-02-07 03:56:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:59 UTC)" (scheduled at 2025-02-07 03:56:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:56:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:56:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:57:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:57:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:57:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:23.420886+00:00 (in 20.001307 seconds)
repositorygcworker stdout | 2025-02-07 03:57:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:03 UTC)" (scheduled at 2025-02-07 03:57:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:57:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:57:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 3, 419788), True, datetime.datetime(2025, 2, 7, 3, 57, 3, 419788), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:57:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:57:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:57:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:57:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:57:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:33.919286+00:00 (in 29.999506 seconds)
buildlogsarchiver stdout | 2025-02-07 03:57:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:33 UTC)" (scheduled at 2025-02-07 03:57:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:57:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 57, 3, 920045), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:57:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:57:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:57:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:57:03,975 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 03:57:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:57:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:59.937899+00:00 (in 55.000729 seconds)
exportactionlogsworker stdout | 2025-02-07 03:57:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:04 UTC)" (scheduled at 2025-02-07 03:57:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:57:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
quotaregistrysizeworker stdout | 2025-02-07 03:57:04,937 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
exportactionlogsworker stdout | 2025-02-07 03:57:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 4, 937473), True, datetime.datetime(2025, 2, 7, 3, 57, 4, 937473), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:57:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:57:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:57:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:57:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:18.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 03:57:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:18 UTC)" (scheduled at 2025-02-07 03:57:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:57:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:57:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 8, 732907), True, datetime.datetime(2025, 2, 7, 3, 57, 8, 732907), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:57:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:57:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:57:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:57:09,029 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:57:09,578 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:57:10,196 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:57:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:57:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:24.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 03:57:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:40 UTC)" (scheduled at 2025-02-07 03:57:10.424250+00:00)
gcworker stdout | 2025-02-07 03:57:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:57:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900330431, None, 1, 0])
gcworker stdout | 2025-02-07 03:57:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:57:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:57:11,353 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:57:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:57:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:41.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 03:57:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:41 UTC)" (scheduled at 2025-02-07 03:57:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:57:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897031836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:57:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:57:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:57:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:57:12,256 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:57:12,352 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:57:13,408 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
namespacegcworker stdout | 2025-02-07 03:57:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:57:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:31.621131+00:00 (in 17.996382 seconds)
namespacegcworker stdout | 2025-02-07 03:57:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:02:13 UTC)" (scheduled at 2025-02-07 03:57:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 03:57:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 57, 13, 625007), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:57:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 57, 13, 625007), True, datetime.datetime(2025, 2, 7, 3, 57, 13, 625007), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:57:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 57, 13, 625007), True, datetime.datetime(2025, 2, 7, 3, 57, 13, 625007), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 3, 57, 13, 625007), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 03:57:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:57:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:02:13 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:13,734 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:57:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:57:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:29.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 03:57:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:29 UTC)" (scheduled at 2025-02-07 03:57:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:57:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:57:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:57:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:57:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:57:14,235 [238] [DEBUG] [app] Starting request: urn:request:8c97d848-b43f-45a5-a7c5-033e728d290e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:14,235 [235] [DEBUG] [app] Starting request: urn:request:74ee800c-a594-49e2-9764-f1cea1d04871 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:57:14,238 [246] [DEBUG] [app] Starting request: urn:request:54e319ba-ccd9-4726-b82c-0325fced8d40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:14,238 [245] [DEBUG] [app] Starting request: urn:request:d3d519f1-c6ca-4ca3-950f-988af9ab82e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:14,239 [246] [DEBUG] [app] Ending request: urn:request:54e319ba-ccd9-4726-b82c-0325fced8d40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54e319ba-ccd9-4726-b82c-0325fced8d40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:57:14,239 [245] [DEBUG] [app] Ending request: urn:request:d3d519f1-c6ca-4ca3-950f-988af9ab82e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d3d519f1-c6ca-4ca3-950f-988af9ab82e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:57:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:57:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:14,242 [238] [DEBUG] [app] Starting request: urn:request:05bec657-12fe-4897-9e61-cf774682a1b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:14,242 [238] [DEBUG] [app] Ending request: urn:request:05bec657-12fe-4897-9e61-cf774682a1b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05bec657-12fe-4897-9e61-cf774682a1b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:57:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:14,242 [237] [DEBUG] [app] Starting request: urn:request:359c7dfb-1df2-4a72-a977-2a9c69b9d367 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:14,243 [237] [DEBUG] [app] Ending request: urn:request:359c7dfb-1df2-4a72-a977-2a9c69b9d367 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:359c7dfb-1df2-4a72-a977-2a9c69b9d367', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:57:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:57:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:14,254 [238] [DEBUG] [app] Ending request: urn:request:8c97d848-b43f-45a5-a7c5-033e728d290e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8c97d848-b43f-45a5-a7c5-033e728d290e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:14,254 [235] [DEBUG] [app] Ending request: urn:request:74ee800c-a594-49e2-9764-f1cea1d04871 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:74ee800c-a594-49e2-9764-f1cea1d04871', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:57:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
globalpromstats stdout | 2025-02-07 03:57:14,468 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 03:57:14,478 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:57:14,534 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:57:14,926 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:57:15,354 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 03:57:15,503 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 03:57:15,508 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 03:57:16,225 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:57:16,228 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:57:16,231 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:57:16,234 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:57:16,239 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:57:17,114 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:57:17,258 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:57:17,262 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:57:17,267 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:57:17,446 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:57:17,618 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:57:18,296 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:57:18,298 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:57:18,305 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:57:18,308 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:57:18,311 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:57:18,313 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:57:18,316 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:57:18,320 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:57:18,326 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:57:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:28.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 03:57:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:28 UTC)" (scheduled at 2025-02-07 03:57:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:57:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:57:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 18, 732879), True, datetime.datetime(2025, 2, 7, 3, 57, 18, 732879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:57:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:57:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:57:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:57:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:57:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:03.419110+00:00 (in 39.997744 seconds)
repositorygcworker stdout | 2025-02-07 03:57:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:23 UTC)" (scheduled at 2025-02-07 03:57:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:57:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:57:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:57:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:57:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:40.424250+00:00 (in 16.000739 seconds)
gcworker stdout | 2025-02-07 03:57:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:54 UTC)" (scheduled at 2025-02-07 03:57:24.423055+00:00)
gcworker stdout | 2025-02-07 03:57:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:57:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:57:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:57:26,660 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:57:28,594 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:57:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:33.814166+00:00 (in 5.081575 seconds)
notificationworker stdout | 2025-02-07 03:57:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:38 UTC)" (scheduled at 2025-02-07 03:57:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:57:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:57:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 28, 732883), True, datetime.datetime(2025, 2, 7, 3, 57, 28, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:57:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:57:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:57:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:57:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:57:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:44.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 03:57:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:44 UTC)" (scheduled at 2025-02-07 03:57:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:57:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:57:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:57:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:57:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:57:29,235 [235] [DEBUG] [app] Starting request: urn:request:9235f361-7820-4064-98c7-716de7681cfb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:29,235 [238] [DEBUG] [app] Starting request: urn:request:d1a61dc4-e372-4c8a-b03d-bb008f40e55c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:57:29,238 [239] [DEBUG] [app] Starting request: urn:request:a4a136e3-9e5b-4af6-b335-a7544e5e779f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:29,238 [246] [DEBUG] [app] Starting request: urn:request:cb5f27e7-204b-472b-a059-fad4e15e8b85 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:29,239 [239] [DEBUG] [app] Ending request: urn:request:a4a136e3-9e5b-4af6-b335-a7544e5e779f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4a136e3-9e5b-4af6-b335-a7544e5e779f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:57:29,239 [246] [DEBUG] [app] Ending request: urn:request:cb5f27e7-204b-472b-a059-fad4e15e8b85 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb5f27e7-204b-472b-a059-fad4e15e8b85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:57:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:57:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:29,242 [238] [DEBUG] [app] Starting request: urn:request:6a81d678-fb79-4567-a504-c338649bdbd1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:29,242 [237] [DEBUG] [app] Starting request: urn:request:469f59ef-7d67-49e5-beae-3050e79aa50c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:29,242 [238] [DEBUG] [app] Ending request: urn:request:6a81d678-fb79-4567-a504-c338649bdbd1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a81d678-fb79-4567-a504-c338649bdbd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:57:29,242 [237] [DEBUG] [app] Ending request: urn:request:469f59ef-7d67-49e5-beae-3050e79aa50c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:469f59ef-7d67-49e5-beae-3050e79aa50c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:57:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:57:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:57:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:29,254 [235] [DEBUG] [app] Ending request: urn:request:9235f361-7820-4064-98c7-716de7681cfb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9235f361-7820-4064-98c7-716de7681cfb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:29,254 [238] [DEBUG] [app] Ending request: urn:request:d1a61dc4-e372-4c8a-b03d-bb008f40e55c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1a61dc4-e372-4c8a-b03d-bb008f40e55c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:57:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 03:57:29,698 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:57:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:57:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:32.625945+00:00 (in 1.004371 seconds)
namespacegcworker stdout | 2025-02-07 03:57:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:31 UTC)" (scheduled at 2025-02-07 03:57:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:57:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:57:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 31, 621855), True, datetime.datetime(2025, 2, 7, 3, 57, 31, 621855), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:57:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:57:31,628 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:57:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:57:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:57:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:31.621131+00:00 (in 58.994706 seconds)
namespacegcworker stdout | 2025-02-07 03:57:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:32 UTC)" (scheduled at 2025-02-07 03:57:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:57:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:57:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:57:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:38.732147+00:00 (in 4.917527 seconds)
notificationworker stdout | 2025-02-07 03:57:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:02:33 UTC)" (scheduled at 2025-02-07 03:57:33.814166+00:00)
notificationworker stdout | 2025-02-07 03:57:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 57, 33, 814872), 'notification/%'])
notificationworker stdout | 2025-02-07 03:57:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 57, 33, 814872), True, datetime.datetime(2025, 2, 7, 3, 57, 33, 814872), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 03:57:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 57, 33, 814872), True, datetime.datetime(2025, 2, 7, 3, 57, 33, 814872), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 3, 57, 33, 814872), 'notification/%'])
notificationworker stdout | 2025-02-07 03:57:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:57:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:02:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:57:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:57:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:03.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 03:57:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:03 UTC)" (scheduled at 2025-02-07 03:57:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:57:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 57, 33, 920001), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:57:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:57:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:57:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:57:34,011 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:57:34,973 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:56.713547+00:00 (in 18.997181 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:37 UTC)" (scheduled at 2025-02-07 03:57:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:57:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:48.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 03:57:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:48 UTC)" (scheduled at 2025-02-07 03:57:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:57:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:57:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 38, 732862), True, datetime.datetime(2025, 2, 7, 3, 57, 38, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:57:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:57:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:57:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:57:39,066 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:57:39,605 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:57:40,219 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:57:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:57:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:54.423055+00:00 (in 13.998308 seconds)
gcworker stdout | 2025-02-07 03:57:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:10 UTC)" (scheduled at 2025-02-07 03:57:40.424250+00:00)
gcworker stdout | 2025-02-07 03:57:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:57:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900360431, None, 1, 0])
gcworker stdout | 2025-02-07 03:57:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:57:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:57:41,371 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:57:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:57:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:11.831521+00:00 (in 29.999538 seconds)
autopruneworker stdout | 2025-02-07 03:57:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:11 UTC)" (scheduled at 2025-02-07 03:57:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:57:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897061836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:57:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:57:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:57:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:57:42,292 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:57:42,388 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:57:43,444 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:43,770 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:57:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:57:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:59.225378+00:00 (in 14.999535 seconds)
quotatotalworker stdout | 2025-02-07 03:57:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:59 UTC)" (scheduled at 2025-02-07 03:57:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:57:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:57:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:57:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:57:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:57:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:57:44,235 [238] [DEBUG] [app] Starting request: urn:request:edf7f8be-2b51-4a83-acda-d5ac4f1dc170 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:44,235 [235] [DEBUG] [app] Starting request: urn:request:84958544-214a-460a-84a6-153ff617a0e6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:57:44,238 [244] [DEBUG] [app] Starting request: urn:request:66e1b493-e65c-4967-85a6-ce25e0268b8f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:44,238 [239] [DEBUG] [app] Starting request: urn:request:6686c2cb-6a24-4154-a155-8ddc4e94dad6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:44,239 [244] [DEBUG] [app] Ending request: urn:request:66e1b493-e65c-4967-85a6-ce25e0268b8f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:66e1b493-e65c-4967-85a6-ce25e0268b8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:57:44,239 [239] [DEBUG] [app] Ending request: urn:request:6686c2cb-6a24-4154-a155-8ddc4e94dad6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6686c2cb-6a24-4154-a155-8ddc4e94dad6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:57:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:57:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:44,243 [237] [DEBUG] [app] Starting request: urn:request:72c0dff1-9978-417d-8e12-c076f53b4929 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:44,243 [236] [DEBUG] [app] Starting request: urn:request:e2385993-c6eb-4dfa-a6ba-38262ff66529 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:44,243 [236] [DEBUG] [app] Ending request: urn:request:e2385993-c6eb-4dfa-a6ba-38262ff66529 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e2385993-c6eb-4dfa-a6ba-38262ff66529', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:57:44,243 [237] [DEBUG] [app] Ending request: urn:request:72c0dff1-9978-417d-8e12-c076f53b4929 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:72c0dff1-9978-417d-8e12-c076f53b4929', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:57:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:44,255 [238] [DEBUG] [app] Ending request: urn:request:edf7f8be-2b51-4a83-acda-d5ac4f1dc170 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:edf7f8be-2b51-4a83-acda-d5ac4f1dc170', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:44,255 [235] [DEBUG] [app] Ending request: urn:request:84958544-214a-460a-84a6-153ff617a0e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84958544-214a-460a-84a6-153ff617a0e6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:57:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
globalpromstats stdout | 2025-02-07 03:57:44,506 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 03:57:44,514 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:57:44,572 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:57:44,962 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:57:45,391 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:57:45,532 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:57:45,541 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:57:46,233 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:57:46,236 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:57:46,239 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:57:46,242 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:57:46,247 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:57:47,151 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:57:47,266 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:57:47,269 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:57:47,273 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:57:47,463 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:57:47,629 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:57:48,303 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:57:48,308 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:57:48,315 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:57:48,318 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:57:48,322 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:57:48,325 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:57:48,328 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:57:48,330 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:57:48,334 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:57:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:50.814677+00:00 (in 2.082087 seconds)
notificationworker stdout | 2025-02-07 03:57:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:58 UTC)" (scheduled at 2025-02-07 03:57:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:57:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:57:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 48, 732915), True, datetime.datetime(2025, 2, 7, 3, 57, 48, 732915), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:57:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:57:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:57:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:57:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:57:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:57:58.732147+00:00 (in 7.917009 seconds)
notificationworker stdout | 2025-02-07 03:57:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:50 UTC)" (scheduled at 2025-02-07 03:57:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:57:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:57:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:57:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:57:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:10.424250+00:00 (in 16.000686 seconds)
gcworker stdout | 2025-02-07 03:57:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:24 UTC)" (scheduled at 2025-02-07 03:57:54.423055+00:00)
gcworker stdout | 2025-02-07 03:57:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:57:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:57:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:57:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:56.118855+00:00 (in 59.999500 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:57:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:56 UTC)" (scheduled at 2025-02-07 03:57:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:57:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:57:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:57:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:57:56,674 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:37.715878+00:00 (in 41.001887 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:56 UTC)" (scheduled at 2025-02-07 03:57:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:57:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:56 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 03:57:58,627 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:57:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:57:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:08.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 03:57:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:08 UTC)" (scheduled at 2025-02-07 03:57:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:57:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:57:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 57, 58, 732858), True, datetime.datetime(2025, 2, 7, 3, 57, 58, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:57:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:57:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:57:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:57:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:57:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:14.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 03:57:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:14 UTC)" (scheduled at 2025-02-07 03:57:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:57:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:57:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:57:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:57:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:57:59,235 [235] [DEBUG] [app] Starting request: urn:request:5edca598-8975-48ed-8f03-11273073b149 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:59,235 [236] [DEBUG] [app] Starting request: urn:request:cd3e9baa-1543-4418-a157-6e8e89b6836d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:57:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:57:59,238 [239] [DEBUG] [app] Starting request: urn:request:e28e4b90-6798-4a72-badb-abcf8b883017 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:59,238 [245] [DEBUG] [app] Starting request: urn:request:f8213d77-120f-4560-8c0e-8649333577f4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:57:59,238 [239] [DEBUG] [app] Ending request: urn:request:e28e4b90-6798-4a72-badb-abcf8b883017 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e28e4b90-6798-4a72-badb-abcf8b883017', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:57:59,238 [245] [DEBUG] [app] Ending request: urn:request:f8213d77-120f-4560-8c0e-8649333577f4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f8213d77-120f-4560-8c0e-8649333577f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:57:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:57:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:57:59,241 [236] [DEBUG] [app] Starting request: urn:request:14ac2c26-eac2-45a9-bb73-d45de198163e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:59,241 [236] [DEBUG] [app] Ending request: urn:request:14ac2c26-eac2-45a9-bb73-d45de198163e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:14ac2c26-eac2-45a9-bb73-d45de198163e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:57:59,241 [238] [DEBUG] [app] Starting request: urn:request:ef68c8e9-db66-45ba-bae1-db7809ba72b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:57:59,241 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:57:59,241 [238] [DEBUG] [app] Ending request: urn:request:ef68c8e9-db66-45ba-bae1-db7809ba72b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef68c8e9-db66-45ba-bae1-db7809ba72b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:57:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:57:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:57:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:57:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:57:59,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:59,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:57:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:57:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:57:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:57:59,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:57:59,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:57:59,252 [236] [DEBUG] [app] Ending request: urn:request:cd3e9baa-1543-4418-a157-6e8e89b6836d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd3e9baa-1543-4418-a157-6e8e89b6836d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:59,253 [235] [DEBUG] [app] Ending request: urn:request:5edca598-8975-48ed-8f03-11273073b149 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5edca598-8975-48ed-8f03-11273073b149', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:57:59,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:57:59,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:57:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:57:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:57:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
queuecleanupworker stdout | 2025-02-07 03:57:59,715 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:57:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:57:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:04.936688+00:00 (in 4.998317 seconds)
exportactionlogsworker stdout | 2025-02-07 03:57:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:59 UTC)" (scheduled at 2025-02-07 03:57:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:57:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:57:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:58:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:58:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:58:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:23.420886+00:00 (in 20.001300 seconds)
repositorygcworker stdout | 2025-02-07 03:58:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:03 UTC)" (scheduled at 2025-02-07 03:58:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:58:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:58:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 3, 419793), True, datetime.datetime(2025, 2, 7, 3, 58, 3, 419793), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:58:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:58:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:58:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:58:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:58:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:33.919286+00:00 (in 29.999528 seconds)
buildlogsarchiver stdout | 2025-02-07 03:58:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:33 UTC)" (scheduled at 2025-02-07 03:58:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:58:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 58, 3, 920028), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:58:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:58:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:58:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:58:04,019 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 03:58:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:58:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:59.937899+00:00 (in 55.000761 seconds)
exportactionlogsworker stdout | 2025-02-07 03:58:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:04 UTC)" (scheduled at 2025-02-07 03:58:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:58:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:58:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 4, 937437), True, datetime.datetime(2025, 2, 7, 3, 58, 4, 937437), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:58:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:58:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:58:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:58:05,009 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 03:58:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:58:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:18.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 03:58:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:18 UTC)" (scheduled at 2025-02-07 03:58:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:58:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:58:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 8, 732883), True, datetime.datetime(2025, 2, 7, 3, 58, 8, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:58:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:58:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:58:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:58:09,097 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:58:09,642 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:58:10,234 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:58:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:58:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:24.423055+00:00 (in 13.998329 seconds)
gcworker stdout | 2025-02-07 03:58:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:40 UTC)" (scheduled at 2025-02-07 03:58:10.424250+00:00)
gcworker stdout | 2025-02-07 03:58:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:58:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900390431, None, 1, 0])
gcworker stdout | 2025-02-07 03:58:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:58:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:58:11,396 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:58:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:58:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 03:58:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:41 UTC)" (scheduled at 2025-02-07 03:58:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:58:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897091836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:58:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:58:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:58:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:58:12,314 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:58:12,424 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:58:13,480 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:13,806 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:58:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:58:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:29.225378+00:00 (in 14.999535 seconds)
quotatotalworker stdout | 2025-02-07 03:58:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:29 UTC)" (scheduled at 2025-02-07 03:58:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:58:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:58:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:58:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:58:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:58:14,235 [235] [DEBUG] [app] Starting request: urn:request:e95207a1-1957-4e74-aafb-358ce693355a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:14,235 [238] [DEBUG] [app] Starting request: urn:request:56ad8f4e-7fcf-4bf1-9a17-dd97d463ae5b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:58:14,238 [246] [DEBUG] [app] Starting request: urn:request:ca3dc610-2528-48d7-af17-bbfb8a078813 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:14,238 [244] [DEBUG] [app] Starting request: urn:request:e19246e0-362d-4644-8018-960b4686395a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:14,239 [246] [DEBUG] [app] Ending request: urn:request:ca3dc610-2528-48d7-af17-bbfb8a078813 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ca3dc610-2528-48d7-af17-bbfb8a078813', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:14,239 [244] [DEBUG] [app] Ending request: urn:request:e19246e0-362d-4644-8018-960b4686395a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e19246e0-362d-4644-8018-960b4686395a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:58:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:58:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:14,241 [235] [DEBUG] [app] Starting request: urn:request:a8b2222f-43fc-4928-afc2-31e6b612bc4c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:58:14,242 [235] [DEBUG] [app] Ending request: urn:request:a8b2222f-43fc-4928-afc2-31e6b612bc4c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a8b2222f-43fc-4928-afc2-31e6b612bc4c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:58:14,242 [236] [DEBUG] [app] Starting request: urn:request:72fb4230-7dad-4729-a574-0bacf951c6eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:58:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:58:14,242 [236] [DEBUG] [app] Ending request: urn:request:72fb4230-7dad-4729-a574-0bacf951c6eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:72fb4230-7dad-4729-a574-0bacf951c6eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:58:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 03:58:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:14,253 [235] [DEBUG] [app] Ending request: urn:request:e95207a1-1957-4e74-aafb-358ce693355a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e95207a1-1957-4e74-aafb-358ce693355a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:58:14,253 [238] [DEBUG] [app] Ending request: urn:request:56ad8f4e-7fcf-4bf1-9a17-dd97d463ae5b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:56ad8f4e-7fcf-4bf1-9a17-dd97d463ae5b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:58:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 03:58:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
globalpromstats stdout | 2025-02-07 03:58:14,538 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 03:58:14,549 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:58:14,608 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:58:14,997 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:58:15,407 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:58:15,568 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:58:15,571 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:58:16,241 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:58:16,244 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:58:16,246 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:58:16,250 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:58:16,253 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:58:17,187 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:58:17,275 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:58:17,279 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:58:17,282 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:58:17,474 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:58:17,662 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:58:18,316 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:58:18,320 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:58:18,326 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:58:18,330 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:58:18,333 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:58:18,335 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:58:18,339 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:58:18,342 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:58:18,345 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:58:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:58:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:28.732147+00:00 (in 9.999491 seconds)
notificationworker stdout | 2025-02-07 03:58:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:28 UTC)" (scheduled at 2025-02-07 03:58:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:58:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:58:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 18, 732981), True, datetime.datetime(2025, 2, 7, 3, 58, 18, 732981), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:58:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:58:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:58:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:58:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:58:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:32.420707+00:00 (in 8.999345 seconds)
repositorygcworker stdout | 2025-02-07 03:58:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:23 UTC)" (scheduled at 2025-02-07 03:58:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:58:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:58:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:58:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:58:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:40.424250+00:00 (in 16.000720 seconds)
gcworker stdout | 2025-02-07 03:58:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:54 UTC)" (scheduled at 2025-02-07 03:58:24.423055+00:00)
gcworker stdout | 2025-02-07 03:58:24,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 03:58:24,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:58:24,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:58:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:58:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:58:26,710 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:58:28,643 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:58:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:58:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:38.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 03:58:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:38 UTC)" (scheduled at 2025-02-07 03:58:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:58:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:58:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 28, 732897), True, datetime.datetime(2025, 2, 7, 3, 58, 28, 732897), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:58:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:58:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:58:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:58:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:58:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:44.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 03:58:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:44 UTC)" (scheduled at 2025-02-07 03:58:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:58:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:58:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:58:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:58:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:58:29,235 [236] [DEBUG] [app] Starting request: urn:request:633d3a2c-c88e-4b7f-b8c7-b535b0dbb8a0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:29,235 [235] [DEBUG] [app] Starting request: urn:request:bb0c4925-6da8-42ff-97e4-4c0621a0e6ac (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:58:29,238 [239] [DEBUG] [app] Starting request: urn:request:bf285571-0259-4b69-b183-2371a6c035bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:29,238 [246] [DEBUG] [app] Starting request: urn:request:0edc9445-d4c7-420c-8b07-767f411fa0fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:29,238 [239] [DEBUG] [app] Ending request: urn:request:bf285571-0259-4b69-b183-2371a6c035bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf285571-0259-4b69-b183-2371a6c035bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:29,238 [246] [DEBUG] [app] Ending request: urn:request:0edc9445-d4c7-420c-8b07-767f411fa0fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0edc9445-d4c7-420c-8b07-767f411fa0fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:58:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:29,241 [238] [DEBUG] [app] Starting request: urn:request:82c46e7e-c9f3-4a79-91a4-53d11a172e6b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:58:29,242 [238] [DEBUG] [app] Ending request: urn:request:82c46e7e-c9f3-4a79-91a4-53d11a172e6b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82c46e7e-c9f3-4a79-91a4-53d11a172e6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:58:29,242 [236] [DEBUG] [app] Starting request: urn:request:28484ab8-f320-45f9-b890-b94718253d13 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:58:29,242 [236] [DEBUG] [app] Ending request: urn:request:28484ab8-f320-45f9-b890-b94718253d13 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28484ab8-f320-45f9-b890-b94718253d13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:58:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:58:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:29,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:29,254 [236] [DEBUG] [app] Ending request: urn:request:633d3a2c-c88e-4b7f-b8c7-b535b0dbb8a0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:633d3a2c-c88e-4b7f-b8c7-b535b0dbb8a0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:58:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:58:29,254 [235] [DEBUG] [app] Ending request: urn:request:bb0c4925-6da8-42ff-97e4-4c0621a0e6ac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb0c4925-6da8-42ff-97e4-4c0621a0e6ac', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 03:58:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 03:58:29,746 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:58:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:58:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:32.625945+00:00 (in 1.004357 seconds)
namespacegcworker stdout | 2025-02-07 03:58:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:31 UTC)" (scheduled at 2025-02-07 03:58:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:58:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:58:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 31, 621897), True, datetime.datetime(2025, 2, 7, 3, 58, 31, 621897), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:58:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:58:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:58:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:58:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:58:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:03.419110+00:00 (in 30.997957 seconds)
repositorygcworker stdout | 2025-02-07 03:58:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:03:32 UTC)" (scheduled at 2025-02-07 03:58:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 03:58:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 58, 32, 421378), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:58:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 58, 32, 421378), True, datetime.datetime(2025, 2, 7, 3, 58, 32, 421378), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:58:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 58, 32, 421378), True, datetime.datetime(2025, 2, 7, 3, 58, 32, 421378), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 3, 58, 32, 421378), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 03:58:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:58:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:03:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:58:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:58:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:31.621131+00:00 (in 58.994734 seconds)
namespacegcworker stdout | 2025-02-07 03:58:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:32 UTC)" (scheduled at 2025-02-07 03:58:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:58:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:58:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:58:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:58:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:03.919286+00:00 (in 29.999539 seconds)
buildlogsarchiver stdout | 2025-02-07 03:58:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:03 UTC)" (scheduled at 2025-02-07 03:58:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:58:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 58, 33, 920004), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:58:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:58:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:58:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:58:34,056 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:58:35,045 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:56.713547+00:00 (in 18.997192 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:37 UTC)" (scheduled at 2025-02-07 03:58:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:58:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:58:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:48.732147+00:00 (in 9.999337 seconds)
notificationworker stdout | 2025-02-07 03:58:38,733 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:48 UTC)" (scheduled at 2025-02-07 03:58:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:58:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:58:38,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 38, 733332), True, datetime.datetime(2025, 2, 7, 3, 58, 38, 733332), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:58:38,743 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:58:38,743 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:58:38,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:58:39,135 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:58:39,661 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:58:40,250 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:58:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:58:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:54.423055+00:00 (in 13.998369 seconds)
gcworker stdout | 2025-02-07 03:58:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:10 UTC)" (scheduled at 2025-02-07 03:58:40.424250+00:00)
gcworker stdout | 2025-02-07 03:58:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:58:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900420431, None, 1, 0])
gcworker stdout | 2025-02-07 03:58:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:58:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:58:41,432 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:58:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:58:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:11.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 03:58:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:11 UTC)" (scheduled at 2025-02-07 03:58:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:58:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897121836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:58:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:58:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:58:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:58:42,324 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:58:42,446 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:58:43,516 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:43,835 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:58:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:58:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:59.225378+00:00 (in 14.999516 seconds)
quotatotalworker stdout | 2025-02-07 03:58:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:59 UTC)" (scheduled at 2025-02-07 03:58:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:58:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:58:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:58:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:58:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:58:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:58:44,235 [235] [DEBUG] [app] Starting request: urn:request:885e7af8-f98e-4700-b2fe-0eaae05dc509 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:44,235 [236] [DEBUG] [app] Starting request: urn:request:229868f9-b80c-488a-bb78-96f8de8fbdea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:58:44,238 [246] [DEBUG] [app] Starting request: urn:request:b31d287f-32b4-482b-a864-3b525cafca63 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:44,238 [245] [DEBUG] [app] Starting request: urn:request:e19c2385-a494-4219-ad6d-3a7bf6252bfb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:44,239 [245] [DEBUG] [app] Ending request: urn:request:e19c2385-a494-4219-ad6d-3a7bf6252bfb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e19c2385-a494-4219-ad6d-3a7bf6252bfb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:44,239 [246] [DEBUG] [app] Ending request: urn:request:b31d287f-32b4-482b-a864-3b525cafca63 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b31d287f-32b4-482b-a864-3b525cafca63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:58:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:44,243 [236] [DEBUG] [app] Starting request: urn:request:a1392739-b376-4e76-af7b-ab9b731a2ab9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:58:44,243 [235] [DEBUG] [app] Starting request: urn:request:64dd8446-5c16-41c7-bff0-dda83b10b7f0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:58:44,243 [236] [DEBUG] [app] Ending request: urn:request:a1392739-b376-4e76-af7b-ab9b731a2ab9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1392739-b376-4e76-af7b-ab9b731a2ab9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:58:44,244 [235] [DEBUG] [app] Ending request: urn:request:64dd8446-5c16-41c7-bff0-dda83b10b7f0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64dd8446-5c16-41c7-bff0-dda83b10b7f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:58:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:58:44,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:44,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:44,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:44,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:44,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:44,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:44,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:44,255 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:44,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:44,256 [235] [DEBUG] [app] Ending request: urn:request:885e7af8-f98e-4700-b2fe-0eaae05dc509 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:885e7af8-f98e-4700-b2fe-0eaae05dc509', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:44,256 [236] [DEBUG] [app] Ending request: urn:request:229868f9-b80c-488a-bb78-96f8de8fbdea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:229868f9-b80c-488a-bb78-96f8de8fbdea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:44,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:58:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:58:44,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:58:44,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
globalpromstats stdout | 2025-02-07 03:58:44,574 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 03:58:44,585 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 03:58:44,641 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:58:45,026 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:58:45,437 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:58:45,578 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:58:45,581 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:58:46,248 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:58:46,251 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:58:46,255 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:58:46,258 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:58:46,260 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:58:47,224 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:58:47,284 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:58:47,287 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:58:47,289 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:58:47,502 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:58:47,698 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:58:48,329 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:58:48,332 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:58:48,336 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:58:48,340 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:58:48,342 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:58:48,345 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:58:48,348 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:58:48,355 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:58:48,358 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:58:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:58:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:50.814677+00:00 (in 2.082011 seconds)
notificationworker stdout | 2025-02-07 03:58:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:58 UTC)" (scheduled at 2025-02-07 03:58:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:58:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:58:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 48, 732953), True, datetime.datetime(2025, 2, 7, 3, 58, 48, 732953), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:58:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:58:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:58:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:58:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:58:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:58:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:58:58.732147+00:00 (in 7.917020 seconds)
notificationworker stdout | 2025-02-07 03:58:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:50 UTC)" (scheduled at 2025-02-07 03:58:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:58:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:58:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:58:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:58:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:10.424250+00:00 (in 16.000735 seconds)
gcworker stdout | 2025-02-07 03:58:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:24 UTC)" (scheduled at 2025-02-07 03:58:54.423055+00:00)
gcworker stdout | 2025-02-07 03:58:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:58:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:58:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:58:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:56.118855+00:00 (in 59.999514 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:58:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:56 UTC)" (scheduled at 2025-02-07 03:58:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:58:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:58:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:58:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:37.715878+00:00 (in 41.001869 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:56 UTC)" (scheduled at 2025-02-07 03:58:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:58:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:58:56,734 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:58:58,651 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:58:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:58:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:08.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 03:58:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:08 UTC)" (scheduled at 2025-02-07 03:58:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:58:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:58:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 58, 58, 732910), True, datetime.datetime(2025, 2, 7, 3, 58, 58, 732910), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:58:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:58:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:58:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:58:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:58:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:14.225378+00:00 (in 14.999574 seconds)
quotatotalworker stdout | 2025-02-07 03:58:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:14 UTC)" (scheduled at 2025-02-07 03:58:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:58:59,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:58:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:58:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:58:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:58:59,235 [235] [DEBUG] [app] Starting request: urn:request:eaa25f27-46b5-4b7e-89e4-b8213aff38d1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:59,235 [237] [DEBUG] [app] Starting request: urn:request:c9451a32-b6c2-4028-ac65-c9fbbd40338c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:58:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:58:59,238 [239] [DEBUG] [app] Starting request: urn:request:c7423251-0440-4ffb-9676-c40f9ff12e3f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:59,238 [245] [DEBUG] [app] Starting request: urn:request:4a6403b2-336e-4e48-a2ee-21f97886c46e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:58:59,239 [239] [DEBUG] [app] Ending request: urn:request:c7423251-0440-4ffb-9676-c40f9ff12e3f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c7423251-0440-4ffb-9676-c40f9ff12e3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:59,239 [245] [DEBUG] [app] Ending request: urn:request:4a6403b2-336e-4e48-a2ee-21f97886c46e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a6403b2-336e-4e48-a2ee-21f97886c46e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:58:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:58:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:58:59,242 [235] [DEBUG] [app] Starting request: urn:request:8b652893-3618-4e1f-b771-593e658fd200 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:58:59,242 [235] [DEBUG] [app] Ending request: urn:request:8b652893-3618-4e1f-b771-593e658fd200 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b652893-3618-4e1f-b771-593e658fd200', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:58:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:58:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:58:59,243 [236] [DEBUG] [app] Starting request: urn:request:8ee72a11-39cf-4da1-ab35-0080762bd387 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:58:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:59,243 [236] [DEBUG] [app] Ending request: urn:request:8ee72a11-39cf-4da1-ab35-0080762bd387 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ee72a11-39cf-4da1-ab35-0080762bd387', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:58:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:58:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:58:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:58:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:58:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:58:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:58:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:58:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:58:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:58:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:58:59,254 [235] [DEBUG] [app] Ending request: urn:request:eaa25f27-46b5-4b7e-89e4-b8213aff38d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eaa25f27-46b5-4b7e-89e4-b8213aff38d1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:58:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:58:59,255 [237] [DEBUG] [app] Ending request: urn:request:c9451a32-b6c2-4028-ac65-c9fbbd40338c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9451a32-b6c2-4028-ac65-c9fbbd40338c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:58:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:58:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:58:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:58:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
queuecleanupworker stdout | 2025-02-07 03:58:59,783 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:58:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:58:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:04.936688+00:00 (in 4.998282 seconds)
exportactionlogsworker stdout | 2025-02-07 03:58:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:59 UTC)" (scheduled at 2025-02-07 03:58:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:58:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:58:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 03:59:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:59:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:59:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:23.420886+00:00 (in 20.001209 seconds)
repositorygcworker stdout | 2025-02-07 03:59:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:03 UTC)" (scheduled at 2025-02-07 03:59:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 03:59:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 03:59:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 3, 419893), True, datetime.datetime(2025, 2, 7, 3, 59, 3, 419893), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 03:59:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 03:59:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 03:59:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:59:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:59:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:33.919286+00:00 (in 29.999533 seconds)
buildlogsarchiver stdout | 2025-02-07 03:59:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:33 UTC)" (scheduled at 2025-02-07 03:59:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:59:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 59, 3, 920008), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:59:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:59:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:59:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:59:04,090 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 03:59:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:59:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:52.937648+00:00 (in 48.000506 seconds)
exportactionlogsworker stdout | 2025-02-07 03:59:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:04 UTC)" (scheduled at 2025-02-07 03:59:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 03:59:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 03:59:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 4, 937445), True, datetime.datetime(2025, 2, 7, 3, 59, 4, 937445), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 03:59:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 03:59:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:59:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:59:05,081 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 03:59:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:59:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:18.732147+00:00 (in 9.999470 seconds)
notificationworker stdout | 2025-02-07 03:59:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:18 UTC)" (scheduled at 2025-02-07 03:59:08.732147+00:00)
notificationworker stdout | 2025-02-07 03:59:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:59:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 8, 733010), True, datetime.datetime(2025, 2, 7, 3, 59, 8, 733010), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:59:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:59:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:59:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:59:09,172 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:59:09,694 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:59:10,274 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:59:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:59:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:24.423055+00:00 (in 13.998345 seconds)
gcworker stdout | 2025-02-07 03:59:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:40 UTC)" (scheduled at 2025-02-07 03:59:10.424250+00:00)
gcworker stdout | 2025-02-07 03:59:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:59:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900450431, None, 1, 0])
gcworker stdout | 2025-02-07 03:59:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:59:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:59:11,458 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:59:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:59:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:41.831521+00:00 (in 29.999580 seconds)
autopruneworker stdout | 2025-02-07 03:59:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:41 UTC)" (scheduled at 2025-02-07 03:59:11.831521+00:00)
autopruneworker stdout | 2025-02-07 03:59:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897151836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:59:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:59:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:59:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:59:12,343 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:59:12,472 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:59:13,530 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:13,841 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:59:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:59:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:29.225378+00:00 (in 14.999528 seconds)
quotatotalworker stdout | 2025-02-07 03:59:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:29 UTC)" (scheduled at 2025-02-07 03:59:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:59:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:59:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:59:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:59:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:59:14,235 [235] [DEBUG] [app] Starting request: urn:request:16e3ff46-aece-4472-969f-f9cc20d244c9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:14,235 [237] [DEBUG] [app] Starting request: urn:request:9a7632d4-47aa-44e6-9876-728cccafbe68 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:59:14,238 [246] [DEBUG] [app] Starting request: urn:request:7c4cb738-4c43-4283-b0be-469fb64e2c53 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:14,239 [245] [DEBUG] [app] Starting request: urn:request:32c4aa71-502b-41eb-905c-f4761e56fdd2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:14,239 [246] [DEBUG] [app] Ending request: urn:request:7c4cb738-4c43-4283-b0be-469fb64e2c53 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7c4cb738-4c43-4283-b0be-469fb64e2c53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:59:14,239 [245] [DEBUG] [app] Ending request: urn:request:32c4aa71-502b-41eb-905c-f4761e56fdd2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:32c4aa71-502b-41eb-905c-f4761e56fdd2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:59:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:59:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 03:59:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:14,242 [237] [DEBUG] [app] Starting request: urn:request:b4c7ca23-4f77-4326-b8a8-9af519507521 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:14,243 [237] [DEBUG] [app] Ending request: urn:request:b4c7ca23-4f77-4326-b8a8-9af519507521 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4c7ca23-4f77-4326-b8a8-9af519507521', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:59:14,243 [238] [DEBUG] [app] Starting request: urn:request:d928d702-d025-4ee5-a45d-f606087441d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:14,243 [238] [DEBUG] [app] Ending request: urn:request:d928d702-d025-4ee5-a45d-f606087441d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d928d702-d025-4ee5-a45d-f606087441d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:59:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:59:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:59:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:59:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:59:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:14,255 [237] [DEBUG] [app] Ending request: urn:request:9a7632d4-47aa-44e6-9876-728cccafbe68 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a7632d4-47aa-44e6-9876-728cccafbe68', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:59:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:59:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:59:14,255 [235] [DEBUG] [app] Ending request: urn:request:16e3ff46-aece-4472-969f-f9cc20d244c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16e3ff46-aece-4472-969f-f9cc20d244c9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:59:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:59:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
servicekey stdout | 2025-02-07 03:59:14,602 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:59:14,606 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:59:14,677 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:59:15,050 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:59:15,473 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:59:15,615 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:59:15,618 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:59:16,256 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:59:16,260 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:59:16,262 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:59:16,265 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:59:16,267 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:59:17,260 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:59:17,292 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:59:17,295 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:59:17,298 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:59:17,512 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:59:17,734 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:59:18,340 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:59:18,343 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:59:18,345 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:59:18,350 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:59:18,352 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:59:18,355 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:59:18,357 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:59:18,365 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:59:18,368 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:59:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:59:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:28.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 03:59:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:28 UTC)" (scheduled at 2025-02-07 03:59:18.732147+00:00)
notificationworker stdout | 2025-02-07 03:59:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:59:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 18, 732905), True, datetime.datetime(2025, 2, 7, 3, 59, 18, 732905), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:59:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:59:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:59:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 03:59:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 03:59:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:03.419110+00:00 (in 39.997709 seconds)
repositorygcworker stdout | 2025-02-07 03:59:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:23 UTC)" (scheduled at 2025-02-07 03:59:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 03:59:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 03:59:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:59:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:59:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:40.424250+00:00 (in 16.000751 seconds)
gcworker stdout | 2025-02-07 03:59:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:54 UTC)" (scheduled at 2025-02-07 03:59:24.423055+00:00)
gcworker stdout | 2025-02-07 03:59:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:59:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 03:59:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:59:26,770 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:59:28,665 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:59:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:59:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:38.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 03:59:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:38 UTC)" (scheduled at 2025-02-07 03:59:28.732147+00:00)
notificationworker stdout | 2025-02-07 03:59:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:59:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 28, 732878), True, datetime.datetime(2025, 2, 7, 3, 59, 28, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:59:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:59:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:59:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:59:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:59:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:44.225378+00:00 (in 14.999532 seconds)
quotatotalworker stdout | 2025-02-07 03:59:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:44 UTC)" (scheduled at 2025-02-07 03:59:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:59:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:59:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:59:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:59:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:59:29,236 [235] [DEBUG] [app] Starting request: urn:request:732b7223-0a3b-4c8a-8697-731048654c95 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:29,236 [237] [DEBUG] [app] Starting request: urn:request:43304034-811f-4aa3-9f98-4b0ef4d967f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:59:29,239 [246] [DEBUG] [app] Starting request: urn:request:f738870b-73bd-4439-adf4-dbd9b097e860 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:29,240 [245] [DEBUG] [app] Starting request: urn:request:2645bc82-b0bd-4eb1-8ef7-ad0280a53850 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:29,240 [246] [DEBUG] [app] Ending request: urn:request:f738870b-73bd-4439-adf4-dbd9b097e860 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f738870b-73bd-4439-adf4-dbd9b097e860', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:59:29,240 [245] [DEBUG] [app] Ending request: urn:request:2645bc82-b0bd-4eb1-8ef7-ad0280a53850 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2645bc82-b0bd-4eb1-8ef7-ad0280a53850', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 03:59:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:59:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:29,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:29,244 [238] [DEBUG] [app] Starting request: urn:request:772dea03-3f2d-471a-8467-e2e769ff94ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:29,244 [236] [DEBUG] [app] Starting request: urn:request:6a68da6b-300c-44db-9d13-7461a7607b8c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:29,244 [238] [DEBUG] [app] Ending request: urn:request:772dea03-3f2d-471a-8467-e2e769ff94ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:772dea03-3f2d-471a-8467-e2e769ff94ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:59:29,244 [236] [DEBUG] [app] Ending request: urn:request:6a68da6b-300c-44db-9d13-7461a7607b8c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a68da6b-300c-44db-9d13-7461a7607b8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:59:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:29,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 03:59:29,245 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:29,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:59:29,246 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:59:29,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:29,246 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:29,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:29,250 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:29,250 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:29,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:29,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:29,255 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:29,256 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:29,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:29,257 [237] [DEBUG] [app] Ending request: urn:request:43304034-811f-4aa3-9f98-4b0ef4d967f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43304034-811f-4aa3-9f98-4b0ef4d967f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:59:29,257 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:59:29,257 [235] [DEBUG] [app] Ending request: urn:request:732b7223-0a3b-4c8a-8697-731048654c95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:732b7223-0a3b-4c8a-8697-731048654c95', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:59:29,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:59:29,258 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.023)
gunicorn-web stdout | 2025-02-07 03:59:29,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 03:59:29,818 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 03:59:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:59:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:32.625945+00:00 (in 1.004358 seconds)
namespacegcworker stdout | 2025-02-07 03:59:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:31 UTC)" (scheduled at 2025-02-07 03:59:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 03:59:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 03:59:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 31, 621869), True, datetime.datetime(2025, 2, 7, 3, 59, 31, 621869), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 03:59:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 03:59:31,628 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 03:59:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 03:59:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 03:59:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:31.621131+00:00 (in 58.994722 seconds)
namespacegcworker stdout | 2025-02-07 03:59:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:32 UTC)" (scheduled at 2025-02-07 03:59:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 03:59:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 03:59:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 03:59:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 03:59:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:03.919286+00:00 (in 29.999541 seconds)
buildlogsarchiver stdout | 2025-02-07 03:59:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:03 UTC)" (scheduled at 2025-02-07 03:59:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 03:59:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 3, 59, 33, 920001), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 03:59:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 03:59:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 03:59:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 03:59:34,118 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 03:59:35,118 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:56.713547+00:00 (in 18.997164 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:37 UTC)" (scheduled at 2025-02-07 03:59:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:59:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:59:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:48.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 03:59:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:48 UTC)" (scheduled at 2025-02-07 03:59:38.732147+00:00)
notificationworker stdout | 2025-02-07 03:59:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:59:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 38, 732850), True, datetime.datetime(2025, 2, 7, 3, 59, 38, 732850), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:59:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:59:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:59:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:59:39,207 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 03:59:39,723 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 03:59:40,310 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 03:59:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:59:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:54.423055+00:00 (in 13.998331 seconds)
gcworker stdout | 2025-02-07 03:59:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:10 UTC)" (scheduled at 2025-02-07 03:59:40.424250+00:00)
gcworker stdout | 2025-02-07 03:59:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 03:59:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900480431, None, 1, 0])
gcworker stdout | 2025-02-07 03:59:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 03:59:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 03:59:41,494 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 03:59:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 03:59:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:11.831521+00:00 (in 29.999556 seconds)
autopruneworker stdout | 2025-02-07 03:59:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:11 UTC)" (scheduled at 2025-02-07 03:59:41.831521+00:00)
autopruneworker stdout | 2025-02-07 03:59:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897181836, None, 1, 0])
autopruneworker stdout | 2025-02-07 03:59:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 03:59:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 03:59:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:59:42,379 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 03:59:42,509 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 03:59:43,567 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:43,878 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 03:59:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:59:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:59.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 03:59:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:59 UTC)" (scheduled at 2025-02-07 03:59:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:59:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:59:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:59:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:59:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 03:59:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:59:44,235 [236] [DEBUG] [app] Starting request: urn:request:8dd3bb3b-fb3b-4ff0-8a11-43a5c1311f1e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:44,235 [235] [DEBUG] [app] Starting request: urn:request:39fa32f0-ae46-4def-955e-5512a20f7c1d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:59:44,238 [239] [DEBUG] [app] Starting request: urn:request:01702946-a629-4ea8-9976-dab6040e6c78 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:44,238 [246] [DEBUG] [app] Starting request: urn:request:055ffa25-050e-44ba-b25d-170e78d177c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:44,239 [239] [DEBUG] [app] Ending request: urn:request:01702946-a629-4ea8-9976-dab6040e6c78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01702946-a629-4ea8-9976-dab6040e6c78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:59:44,239 [246] [DEBUG] [app] Ending request: urn:request:055ffa25-050e-44ba-b25d-170e78d177c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:055ffa25-050e-44ba-b25d-170e78d177c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:59:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 03:59:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:44,242 [235] [DEBUG] [app] Starting request: urn:request:b9d31fb7-86a8-48ab-b5e3-56314387ce61 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:44,242 [238] [DEBUG] [app] Starting request: urn:request:c61b73a2-03c7-4526-8564-9b2ffb7301b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:44,242 [235] [DEBUG] [app] Ending request: urn:request:b9d31fb7-86a8-48ab-b5e3-56314387ce61 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9d31fb7-86a8-48ab-b5e3-56314387ce61', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:59:44,242 [238] [DEBUG] [app] Ending request: urn:request:c61b73a2-03c7-4526-8564-9b2ffb7301b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c61b73a2-03c7-4526-8564-9b2ffb7301b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:59:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:59:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:59:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:59:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:59:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:44,254 [235] [DEBUG] [app] Ending request: urn:request:39fa32f0-ae46-4def-955e-5512a20f7c1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39fa32f0-ae46-4def-955e-5512a20f7c1d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:59:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:59:44,254 [236] [DEBUG] [app] Ending request: urn:request:8dd3bb3b-fb3b-4ff0-8a11-43a5c1311f1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8dd3bb3b-fb3b-4ff0-8a11-43a5c1311f1e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:59:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:59:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 03:59:44,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
servicekey stdout | 2025-02-07 03:59:44,609 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 03:59:44,634 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 03:59:44,700 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 03:59:45,079 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 03:59:45,483 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 03:59:45,640 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 03:59:45,653 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 03:59:46,264 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 03:59:46,267 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 03:59:46,271 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 03:59:46,274 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 03:59:46,277 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
namespacegcworker stdout | 2025-02-07 03:59:47,286 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-secscan stdout | 2025-02-07 03:59:47,301 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 03:59:47,305 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 03:59:47,309 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
buildlogsarchiver stdout | 2025-02-07 03:59:47,531 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 03:59:47,771 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 03:59:48,351 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 03:59:48,354 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 03:59:48,356 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 03:59:48,359 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 03:59:48,362 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 03:59:48,365 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 03:59:48,368 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 03:59:48,374 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 03:59:48,377 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 03:59:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:59:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:50.814677+00:00 (in 2.082102 seconds)
notificationworker stdout | 2025-02-07 03:59:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:58 UTC)" (scheduled at 2025-02-07 03:59:48.732147+00:00)
notificationworker stdout | 2025-02-07 03:59:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:59:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 48, 732862), True, datetime.datetime(2025, 2, 7, 3, 59, 48, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:59:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:59:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:59:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 03:59:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 03:59:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:59:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:58.732147+00:00 (in 7.917037 seconds)
notificationworker stdout | 2025-02-07 03:59:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:50 UTC)" (scheduled at 2025-02-07 03:59:50.814677+00:00)
notificationworker stdout | 2025-02-07 03:59:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 03:59:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 03:59:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:59:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 03:59:59.937899+00:00 (in 6.999786 seconds)
exportactionlogsworker stdout | 2025-02-07 03:59:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:04:52 UTC)" (scheduled at 2025-02-07 03:59:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 03:59:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 3, 59, 52, 938388), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:59:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 59, 52, 938388), True, datetime.datetime(2025, 2, 7, 3, 59, 52, 938388), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:59:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 3, 59, 52, 938388), True, datetime.datetime(2025, 2, 7, 3, 59, 52, 938388), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 3, 59, 52, 938388), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 03:59:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 03:59:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:04:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 03:59:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 03:59:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:10.424250+00:00 (in 16.000710 seconds)
gcworker stdout | 2025-02-07 03:59:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:24 UTC)" (scheduled at 2025-02-07 03:59:54.423055+00:00)
gcworker stdout | 2025-02-07 03:59:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 03:59:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 03:59:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 03:59:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:56.118855+00:00 (in 59.999501 seconds)
quotaregistrysizeworker stdout | 2025-02-07 03:59:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:56 UTC)" (scheduled at 2025-02-07 03:59:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 03:59:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 03:59:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 03:59:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:37.715878+00:00 (in 41.001837 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:56 UTC)" (scheduled at 2025-02-07 03:59:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 03:59:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 03:59:56,807 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 03:59:58,683 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 03:59:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 03:59:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:08.732147+00:00 (in 9.999575 seconds)
notificationworker stdout | 2025-02-07 03:59:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:08 UTC)" (scheduled at 2025-02-07 03:59:58.732147+00:00)
notificationworker stdout | 2025-02-07 03:59:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 03:59:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 3, 59, 58, 732843), True, datetime.datetime(2025, 2, 7, 3, 59, 58, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 03:59:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 03:59:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 03:59:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 03:59:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 03:59:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:14.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 03:59:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:14 UTC)" (scheduled at 2025-02-07 03:59:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 03:59:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 03:59:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 03:59:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 03:59:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 03:59:59,236 [237] [DEBUG] [app] Starting request: urn:request:45e37224-7ac9-4f63-a8d3-ee538b2253dd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:59,236 [238] [DEBUG] [app] Starting request: urn:request:c2c4a6ff-3fcd-452e-848b-fcb8cd2df515 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 03:59:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 03:59:59,239 [239] [DEBUG] [app] Starting request: urn:request:fd3c0156-dee2-4826-ab85-c5a49a8df323 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:59,239 [245] [DEBUG] [app] Starting request: urn:request:4800cf34-0e70-4e0d-a9bf-10bbba1a07d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 03:59:59,239 [239] [DEBUG] [app] Ending request: urn:request:fd3c0156-dee2-4826-ab85-c5a49a8df323 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd3c0156-dee2-4826-ab85-c5a49a8df323', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:59:59,239 [245] [DEBUG] [app] Ending request: urn:request:4800cf34-0e70-4e0d-a9bf-10bbba1a07d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4800cf34-0e70-4e0d-a9bf-10bbba1a07d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 03:59:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 03:59:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 03:59:59,242 [237] [DEBUG] [app] Starting request: urn:request:c5c0efe6-dc73-47a6-948f-68d7a8d3fac8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:59,242 [237] [DEBUG] [app] Ending request: urn:request:c5c0efe6-dc73-47a6-948f-68d7a8d3fac8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5c0efe6-dc73-47a6-948f-68d7a8d3fac8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 03:59:59,242 [236] [DEBUG] [app] Starting request: urn:request:f1584cfe-b5c5-4e60-9331-bec7090762e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 03:59:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:59,243 [236] [DEBUG] [app] Ending request: urn:request:f1584cfe-b5c5-4e60-9331-bec7090762e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f1584cfe-b5c5-4e60-9331-bec7090762e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 03:59:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:03:59:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 03:59:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 03:59:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:03:59:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 03:59:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 03:59:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 03:59:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 03:59:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 03:59:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 03:59:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 03:59:59,254 [238] [DEBUG] [app] Ending request: urn:request:c2c4a6ff-3fcd-452e-848b-fcb8cd2df515 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2c4a6ff-3fcd-452e-848b-fcb8cd2df515', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:59:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 03:59:59,255 [237] [DEBUG] [app] Ending request: urn:request:45e37224-7ac9-4f63-a8d3-ee538b2253dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:45e37224-7ac9-4f63-a8d3-ee538b2253dd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 03:59:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 03:59:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 03:59:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:03:59:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:03:59:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 03:59:59,854 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 03:59:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 03:59:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:04.936688+00:00 (in 4.998315 seconds)
exportactionlogsworker stdout | 2025-02-07 03:59:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:59 UTC)" (scheduled at 2025-02-07 03:59:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 03:59:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 03:59:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:00:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:00:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:00:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:23.420886+00:00 (in 20.001307 seconds)
repositorygcworker stdout | 2025-02-07 04:00:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:03 UTC)" (scheduled at 2025-02-07 04:00:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:00:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:00:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 3, 419782), True, datetime.datetime(2025, 2, 7, 4, 0, 3, 419782), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:00:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:00:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:00:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:00:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:00:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:33.919286+00:00 (in 29.999549 seconds)
buildlogsarchiver stdout | 2025-02-07 04:00:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:33 UTC)" (scheduled at 2025-02-07 04:00:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:00:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 0, 3, 919986), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:00:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:00:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:00:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:00:04,154 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:00:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:00:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:59.937899+00:00 (in 55.000598 seconds)
exportactionlogsworker stdout | 2025-02-07 04:00:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:04 UTC)" (scheduled at 2025-02-07 04:00:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:00:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:00:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 4, 937656), True, datetime.datetime(2025, 2, 7, 4, 0, 4, 937656), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:00:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:00:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:00:04,946 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:00:05,131 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:00:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:00:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:18.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 04:00:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:18 UTC)" (scheduled at 2025-02-07 04:00:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:00:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:00:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 8, 732873), True, datetime.datetime(2025, 2, 7, 4, 0, 8, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:00:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:00:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:00:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:00:09,240 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:00:09,759 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:00:10,347 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 04:00:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:00:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:24.423055+00:00 (in 13.998356 seconds)
gcworker stdout | 2025-02-07 04:00:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:40 UTC)" (scheduled at 2025-02-07 04:00:10.424250+00:00)
gcworker stdout | 2025-02-07 04:00:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:00:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900510430, None, 1, 0])
gcworker stdout | 2025-02-07 04:00:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:00:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:00:11,518 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:00:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:00:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:41.831521+00:00 (in 29.999545 seconds)
autopruneworker stdout | 2025-02-07 04:00:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:41 UTC)" (scheduled at 2025-02-07 04:00:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:00:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897211836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:00:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:00:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:00:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:00:12,410 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:00:12,545 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:00:13,603 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:13,914 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:00:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:00:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:29.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 04:00:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:29 UTC)" (scheduled at 2025-02-07 04:00:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:00:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:00:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:00:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:00:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:00:14,235 [235] [DEBUG] [app] Starting request: urn:request:ece08728-e85e-4399-9b35-3a0411cb577c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:14,235 [238] [DEBUG] [app] Starting request: urn:request:2504703c-2694-4ec8-9706-e22187d391e3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:00:14,238 [239] [DEBUG] [app] Starting request: urn:request:7819ba55-73aa-4cd6-9656-cec4f3ee242d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:14,238 [246] [DEBUG] [app] Starting request: urn:request:7a846c23-5342-4532-8b37-adf5fb9a2fcc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:14,238 [239] [DEBUG] [app] Ending request: urn:request:7819ba55-73aa-4cd6-9656-cec4f3ee242d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7819ba55-73aa-4cd6-9656-cec4f3ee242d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:00:14,238 [246] [DEBUG] [app] Ending request: urn:request:7a846c23-5342-4532-8b37-adf5fb9a2fcc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7a846c23-5342-4532-8b37-adf5fb9a2fcc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:00:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:00:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:00:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:14,241 [237] [DEBUG] [app] Starting request: urn:request:ced9cd21-daa7-4aa3-9d4f-7d3269759283 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:14,241 [235] [DEBUG] [app] Starting request: urn:request:5f81b11d-0374-4f76-af14-6252067b5820 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:14,242 [237] [DEBUG] [app] Ending request: urn:request:ced9cd21-daa7-4aa3-9d4f-7d3269759283 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ced9cd21-daa7-4aa3-9d4f-7d3269759283', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:00:14,242 [235] [DEBUG] [app] Ending request: urn:request:5f81b11d-0374-4f76-af14-6252067b5820 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f81b11d-0374-4f76-af14-6252067b5820', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:00:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:00:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:00:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:14,253 [238] [DEBUG] [app] Ending request: urn:request:2504703c-2694-4ec8-9706-e22187d391e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2504703c-2694-4ec8-9706-e22187d391e3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:14,253 [235] [DEBUG] [app] Ending request: urn:request:ece08728-e85e-4399-9b35-3a0411cb577c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ece08728-e85e-4399-9b35-3a0411cb577c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 04:00:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
servicekey stdout | 2025-02-07 04:00:14,646 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:00:14,658 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:00:14,737 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:00:15,114 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:00:15,493 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 04:00:15,677 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 04:00:15,688 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 04:00:16,273 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:00:16,277 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:00:16,280 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:00:16,284 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:00:16,288 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:00:17,308 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:00:17,312 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:00:17,315 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:00:17,320 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:00:17,567 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:00:17,807 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:00:18,362 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:00:18,366 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:00:18,369 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:00:18,373 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:00:18,376 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:00:18,379 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:00:18,382 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:00:18,385 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:00:18,388 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:00:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:00:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:28.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 04:00:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:28 UTC)" (scheduled at 2025-02-07 04:00:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:00:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:00:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 18, 732901), True, datetime.datetime(2025, 2, 7, 4, 0, 18, 732901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:00:18,748 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:00:18,748 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:00:18,748 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:00:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:00:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:03.419110+00:00 (in 39.997746 seconds)
repositorygcworker stdout | 2025-02-07 04:00:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:23 UTC)" (scheduled at 2025-02-07 04:00:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:00:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:00:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:00:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:00:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:40.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 04:00:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:54 UTC)" (scheduled at 2025-02-07 04:00:24.423055+00:00)
gcworker stdout | 2025-02-07 04:00:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:00:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:00:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:00:26,840 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:00:28,706 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 04:00:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:00:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:38.732147+00:00 (in 9.999517 seconds)
notificationworker stdout | 2025-02-07 04:00:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:38 UTC)" (scheduled at 2025-02-07 04:00:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:00:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:00:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 28, 732933), True, datetime.datetime(2025, 2, 7, 4, 0, 28, 732933), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:00:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:00:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:00:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:00:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:00:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:44.225378+00:00 (in 14.999550 seconds)
quotatotalworker stdout | 2025-02-07 04:00:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:44 UTC)" (scheduled at 2025-02-07 04:00:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:00:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:00:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:00:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:00:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:00:29,236 [238] [DEBUG] [app] Starting request: urn:request:a027ce9f-0ddc-4c81-a6a7-9f2e1a6438c9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:29,236 [235] [DEBUG] [app] Starting request: urn:request:cea276a0-bd2e-428d-8dc1-485104e1418e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:00:29,238 [246] [DEBUG] [app] Starting request: urn:request:93413145-3adc-4046-bbe7-15e89eb70c62 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:29,238 [240] [DEBUG] [app] Starting request: urn:request:f92bf302-64c6-4513-a97d-f84dcadc37f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:29,239 [240] [DEBUG] [app] Ending request: urn:request:f92bf302-64c6-4513-a97d-f84dcadc37f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f92bf302-64c6-4513-a97d-f84dcadc37f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:00:29,239 [246] [DEBUG] [app] Ending request: urn:request:93413145-3adc-4046-bbe7-15e89eb70c62 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93413145-3adc-4046-bbe7-15e89eb70c62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:00:29,239 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:00:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:00:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:29,242 [237] [DEBUG] [app] Starting request: urn:request:ef3fcaf6-158f-4b09-b684-e417b10c618a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:29,242 [236] [DEBUG] [app] Starting request: urn:request:33758dca-fd74-4fea-847c-1c925e46f606 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:29,242 [236] [DEBUG] [app] Ending request: urn:request:33758dca-fd74-4fea-847c-1c925e46f606 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:33758dca-fd74-4fea-847c-1c925e46f606', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:00:29,242 [237] [DEBUG] [app] Ending request: urn:request:ef3fcaf6-158f-4b09-b684-e417b10c618a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef3fcaf6-158f-4b09-b684-e417b10c618a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:00:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:00:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:00:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:00:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:29,254 [238] [DEBUG] [app] Ending request: urn:request:a027ce9f-0ddc-4c81-a6a7-9f2e1a6438c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a027ce9f-0ddc-4c81-a6a7-9f2e1a6438c9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:29,254 [235] [DEBUG] [app] Ending request: urn:request:cea276a0-bd2e-428d-8dc1-485104e1418e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cea276a0-bd2e-428d-8dc1-485104e1418e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:00:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
queuecleanupworker stdout | 2025-02-07 04:00:29,887 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:00:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:00:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:32.625945+00:00 (in 1.004336 seconds)
namespacegcworker stdout | 2025-02-07 04:00:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:31 UTC)" (scheduled at 2025-02-07 04:00:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:00:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:00:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 31, 621934), True, datetime.datetime(2025, 2, 7, 4, 0, 31, 621934), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:00:31,630 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:00:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:00:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:00:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:00:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:31.621131+00:00 (in 58.994693 seconds)
namespacegcworker stdout | 2025-02-07 04:00:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:32 UTC)" (scheduled at 2025-02-07 04:00:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:00:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:00:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:00:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:00:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:03.919286+00:00 (in 29.999522 seconds)
buildlogsarchiver stdout | 2025-02-07 04:00:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:03 UTC)" (scheduled at 2025-02-07 04:00:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:00:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 0, 33, 920054), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:00:33,928 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:00:33,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:00:33,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:00:34,190 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:00:35,156 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:56.713547+00:00 (in 18.997190 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:37 UTC)" (scheduled at 2025-02-07 04:00:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:00:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:00:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:48.732147+00:00 (in 9.999495 seconds)
notificationworker stdout | 2025-02-07 04:00:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:48 UTC)" (scheduled at 2025-02-07 04:00:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:00:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:00:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 38, 732960), True, datetime.datetime(2025, 2, 7, 4, 0, 38, 732960), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:00:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:00:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:00:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:00:39,274 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:00:39,788 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:00:40,383 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 04:00:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:00:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:54.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 04:00:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:10 UTC)" (scheduled at 2025-02-07 04:00:40.424250+00:00)
gcworker stdout | 2025-02-07 04:00:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:00:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900540431, None, 1, 0])
gcworker stdout | 2025-02-07 04:00:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:00:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:00:41,554 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:00:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:00:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:11.831521+00:00 (in 29.999545 seconds)
autopruneworker stdout | 2025-02-07 04:00:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:11 UTC)" (scheduled at 2025-02-07 04:00:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:00:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897241836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:00:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:00:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:00:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:00:42,446 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:00:42,578 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:00:43,634 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:43,930 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:00:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:00:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:59.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 04:00:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:59 UTC)" (scheduled at 2025-02-07 04:00:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:00:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:00:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:00:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:00:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:00:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:00:44,235 [235] [DEBUG] [app] Starting request: urn:request:9ed3f28c-6955-4b28-9a31-a93ff3d11f56 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:44,235 [238] [DEBUG] [app] Starting request: urn:request:e85f68ec-28f7-4414-ad73-156a20e0bc96 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:00:44,238 [246] [DEBUG] [app] Starting request: urn:request:b9016c9e-1766-429a-aa90-c4eea76dcbe5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:44,238 [239] [DEBUG] [app] Starting request: urn:request:02a6706c-f0f9-4884-b9c9-2e5865c088cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:44,238 [246] [DEBUG] [app] Ending request: urn:request:b9016c9e-1766-429a-aa90-c4eea76dcbe5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b9016c9e-1766-429a-aa90-c4eea76dcbe5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:00:44,238 [239] [DEBUG] [app] Ending request: urn:request:02a6706c-f0f9-4884-b9c9-2e5865c088cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02a6706c-f0f9-4884-b9c9-2e5865c088cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:00:44,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:00:44,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:00:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:44,241 [238] [DEBUG] [app] Starting request: urn:request:5053881f-8952-4fe1-a40a-163449bb8141 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:44,241 [238] [DEBUG] [app] Ending request: urn:request:5053881f-8952-4fe1-a40a-163449bb8141 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5053881f-8952-4fe1-a40a-163449bb8141', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:00:44,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:00:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:44,242 [237] [DEBUG] [app] Starting request: urn:request:6f113a60-4e23-4ff7-afbb-c84b465268b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:44,242 [237] [DEBUG] [app] Ending request: urn:request:6f113a60-4e23-4ff7-afbb-c84b465268b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f113a60-4e23-4ff7-afbb-c84b465268b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:00:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:00:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:44,245 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:44,245 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:44,249 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:44,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:44,252 [238] [DEBUG] [app] Ending request: urn:request:e85f68ec-28f7-4414-ad73-156a20e0bc96 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e85f68ec-28f7-4414-ad73-156a20e0bc96', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:44,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:44,252 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:00:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:44,254 [235] [DEBUG] [app] Ending request: urn:request:9ed3f28c-6955-4b28-9a31-a93ff3d11f56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ed3f28c-6955-4b28-9a31-a93ff3d11f56', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
globalpromstats stdout | 2025-02-07 04:00:44,667 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:00:44,671 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:00:44,762 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:00:45,145 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:00:45,510 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 04:00:45,708 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 04:00:45,721 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 04:00:46,281 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:00:46,285 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:00:46,288 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:00:46,292 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:00:46,295 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:00:47,317 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:00:47,320 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:00:47,323 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:00:47,343 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:00:47,580 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:00:47,842 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:00:48,372 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:00:48,376 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:00:48,379 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:00:48,382 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:00:48,384 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:00:48,387 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:00:48,393 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:00:48,396 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:00:48,398 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:00:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:00:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:50.814677+00:00 (in 2.082053 seconds)
notificationworker stdout | 2025-02-07 04:00:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:58 UTC)" (scheduled at 2025-02-07 04:00:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:00:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:00:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 48, 732902), True, datetime.datetime(2025, 2, 7, 4, 0, 48, 732902), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:00:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:00:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:00:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:00:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:00:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:00:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:00:58.732147+00:00 (in 7.916969 seconds)
notificationworker stdout | 2025-02-07 04:00:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:50 UTC)" (scheduled at 2025-02-07 04:00:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:00:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:00:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:00:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:00:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:10.424250+00:00 (in 16.000757 seconds)
gcworker stdout | 2025-02-07 04:00:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:24 UTC)" (scheduled at 2025-02-07 04:00:54.423055+00:00)
gcworker stdout | 2025-02-07 04:00:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:00:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:00:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:00:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:56.118855+00:00 (in 59.999485 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:00:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:56 UTC)" (scheduled at 2025-02-07 04:00:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:00:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:00:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:00:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:37.715878+00:00 (in 41.001849 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:56 UTC)" (scheduled at 2025-02-07 04:00:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:00:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:00:56,876 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:00:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:00:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:08.732147+00:00 (in 9.999468 seconds)
notificationworker stdout | 2025-02-07 04:00:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:08 UTC)" (scheduled at 2025-02-07 04:00:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:00:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:00:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 0, 58, 732980), True, datetime.datetime(2025, 2, 7, 4, 0, 58, 732980), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:00:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:00:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:00:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:00:58,742 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:00:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:00:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:14.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 04:00:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:14 UTC)" (scheduled at 2025-02-07 04:00:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:00:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:00:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:00:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:00:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:00:59,236 [235] [DEBUG] [app] Starting request: urn:request:1cf22a98-f34a-45d7-8a7d-a915ad099cbb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:59,236 [238] [DEBUG] [app] Starting request: urn:request:1e2ba0b2-ad0c-4d92-9982-2a03e7a2334c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:00:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:59,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:00:59,240 [239] [DEBUG] [app] Starting request: urn:request:523be381-0192-4127-97bb-9d6106be08ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:59,240 [243] [DEBUG] [app] Starting request: urn:request:541e9ab1-0c89-415d-80a3-5a4d10c58177 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:00:59,240 [243] [DEBUG] [app] Ending request: urn:request:541e9ab1-0c89-415d-80a3-5a4d10c58177 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:541e9ab1-0c89-415d-80a3-5a4d10c58177', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:00:59,240 [239] [DEBUG] [app] Ending request: urn:request:523be381-0192-4127-97bb-9d6106be08ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:523be381-0192-4127-97bb-9d6106be08ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:00:59,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:00:59,241 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:00:59,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:59,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:00:59,244 [236] [DEBUG] [app] Starting request: urn:request:6c4e8011-a0e2-4e60-834e-30427517cd23 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:59,244 [237] [DEBUG] [app] Starting request: urn:request:4776116c-8af9-4085-8c46-aed35274a7a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:00:59,244 [236] [DEBUG] [app] Ending request: urn:request:6c4e8011-a0e2-4e60-834e-30427517cd23 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6c4e8011-a0e2-4e60-834e-30427517cd23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:00:59,244 [237] [DEBUG] [app] Ending request: urn:request:4776116c-8af9-4085-8c46-aed35274a7a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4776116c-8af9-4085-8c46-aed35274a7a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:00:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:00:59,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:00:59,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:00:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:00:59,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:59,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:00:59,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:59,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:00:59,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:59,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:00:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:59,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:00:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:59,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:00:59,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:59,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:00:59,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:59,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:00:59,256 [238] [DEBUG] [app] Ending request: urn:request:1e2ba0b2-ad0c-4d92-9982-2a03e7a2334c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e2ba0b2-ad0c-4d92-9982-2a03e7a2334c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:59,256 [235] [DEBUG] [app] Ending request: urn:request:1cf22a98-f34a-45d7-8a7d-a915ad099cbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1cf22a98-f34a-45d7-8a7d-a915ad099cbb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:00:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:00:59,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:00:59,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:00:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:00:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
queuecleanupworker stdout | 2025-02-07 04:00:59,918 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 04:00:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:00:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:04.936688+00:00 (in 4.998338 seconds)
exportactionlogsworker stdout | 2025-02-07 04:00:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:59 UTC)" (scheduled at 2025-02-07 04:00:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:00:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:00:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:01:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:01:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:01:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:23.420886+00:00 (in 20.001303 seconds)
repositorygcworker stdout | 2025-02-07 04:01:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:03 UTC)" (scheduled at 2025-02-07 04:01:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:01:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:01:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 3, 419864), True, datetime.datetime(2025, 2, 7, 4, 1, 3, 419864), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:01:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:01:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:01:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:01:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:01:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:33.919286+00:00 (in 29.999498 seconds)
buildlogsarchiver stdout | 2025-02-07 04:01:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:33 UTC)" (scheduled at 2025-02-07 04:01:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:01:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 1, 3, 920077), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:01:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:01:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:01:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:01:04,216 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:01:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:01:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:59.937899+00:00 (in 55.000780 seconds)
exportactionlogsworker stdout | 2025-02-07 04:01:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:04 UTC)" (scheduled at 2025-02-07 04:01:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:01:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:01:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 4, 937420), True, datetime.datetime(2025, 2, 7, 4, 1, 4, 937420), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:01:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:01:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:01:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:01:05,192 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:01:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:01:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:18.732147+00:00 (in 9.999487 seconds)
notificationworker stdout | 2025-02-07 04:01:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:18 UTC)" (scheduled at 2025-02-07 04:01:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:01:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:01:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 8, 732974), True, datetime.datetime(2025, 2, 7, 4, 1, 8, 732974), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:01:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:01:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:01:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:01:09,310 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:01:09,796 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:01:10,394 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 04:01:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:01:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:24.423055+00:00 (in 13.998389 seconds)
gcworker stdout | 2025-02-07 04:01:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:40 UTC)" (scheduled at 2025-02-07 04:01:10.424250+00:00)
gcworker stdout | 2025-02-07 04:01:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:01:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900570431, None, 1, 0])
gcworker stdout | 2025-02-07 04:01:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:01:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:01:11,578 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:01:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:01:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 04:01:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:41 UTC)" (scheduled at 2025-02-07 04:01:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:01:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897271836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:01:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:01:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:01:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:01:12,473 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:01:12,590 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:01:13,641 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:13,968 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:01:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:01:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:29.225378+00:00 (in 14.999538 seconds)
quotatotalworker stdout | 2025-02-07 04:01:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:29 UTC)" (scheduled at 2025-02-07 04:01:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:01:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:01:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:01:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:01:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:01:14,236 [236] [DEBUG] [app] Starting request: urn:request:104e23a0-bdd3-47ad-83a5-d47184c3769c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:14,236 [235] [DEBUG] [app] Starting request: urn:request:8f813bd8-6efc-4668-a23e-41d8b8335bc1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:14,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:01:14,240 [246] [DEBUG] [app] Starting request: urn:request:8d94fa73-82a8-46d9-b402-173799eed1ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:14,240 [239] [DEBUG] [app] Starting request: urn:request:44ea3542-2b55-4ebe-82da-e0b3e3b68f1f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:14,240 [246] [DEBUG] [app] Ending request: urn:request:8d94fa73-82a8-46d9-b402-173799eed1ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d94fa73-82a8-46d9-b402-173799eed1ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:01:14,240 [239] [DEBUG] [app] Ending request: urn:request:44ea3542-2b55-4ebe-82da-e0b3e3b68f1f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44ea3542-2b55-4ebe-82da-e0b3e3b68f1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:01:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:01:14,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:14,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:14,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:01:14,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:14,243 [236] [DEBUG] [app] Starting request: urn:request:52055048-fd0e-43a9-8908-703c26c46e8f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:14,243 [236] [DEBUG] [app] Ending request: urn:request:52055048-fd0e-43a9-8908-703c26c46e8f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:52055048-fd0e-43a9-8908-703c26c46e8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:01:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:14,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:14,244 [237] [DEBUG] [app] Starting request: urn:request:0574ec62-417f-4525-b062-73f23acc0be1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:14,244 [237] [DEBUG] [app] Ending request: urn:request:0574ec62-417f-4525-b062-73f23acc0be1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0574ec62-417f-4525-b062-73f23acc0be1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:01:14,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:14,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:01:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:14,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:14,255 [236] [DEBUG] [app] Ending request: urn:request:104e23a0-bdd3-47ad-83a5-d47184c3769c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:104e23a0-bdd3-47ad-83a5-d47184c3769c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:01:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:01:14,256 [235] [DEBUG] [app] Ending request: urn:request:8f813bd8-6efc-4668-a23e-41d8b8335bc1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f813bd8-6efc-4668-a23e-41d8b8335bc1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:01:14,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
servicekey stdout | 2025-02-07 04:01:14,690 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:01:14,703 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:01:14,798 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:01:15,180 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:01:15,546 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 04:01:15,745 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 04:01:15,756 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 04:01:16,290 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:01:16,293 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:01:16,297 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:01:16,300 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:01:16,302 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:01:17,324 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:01:17,327 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:01:17,330 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:01:17,364 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:01:17,616 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:01:17,866 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:01:18,381 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:01:18,384 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:01:18,389 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:01:18,393 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:01:18,396 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:01:18,398 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:01:18,402 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:01:18,406 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:01:18,408 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:01:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:01:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:28.732147+00:00 (in 9.999516 seconds)
notificationworker stdout | 2025-02-07 04:01:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:28 UTC)" (scheduled at 2025-02-07 04:01:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:01:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:01:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 18, 732916), True, datetime.datetime(2025, 2, 7, 4, 1, 18, 732916), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:01:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:01:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:01:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:01:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:01:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:03.419110+00:00 (in 39.997726 seconds)
repositorygcworker stdout | 2025-02-07 04:01:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:23 UTC)" (scheduled at 2025-02-07 04:01:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:01:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:01:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:01:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:01:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:40.424250+00:00 (in 16.000736 seconds)
gcworker stdout | 2025-02-07 04:01:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:54 UTC)" (scheduled at 2025-02-07 04:01:24.423055+00:00)
gcworker stdout | 2025-02-07 04:01:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:01:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:01:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:01:26,884 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:01:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:01:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:38.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 04:01:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:38 UTC)" (scheduled at 2025-02-07 04:01:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:01:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:01:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 28, 732849), True, datetime.datetime(2025, 2, 7, 4, 1, 28, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:01:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:01:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:01:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:01:28,764 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:01:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:01:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:44.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:01:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:44 UTC)" (scheduled at 2025-02-07 04:01:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:01:29,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:01:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:01:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:01:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:01:29,235 [237] [DEBUG] [app] Starting request: urn:request:27047516-6fa6-4955-9c41-1c780778adfe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:29,235 [235] [DEBUG] [app] Starting request: urn:request:97c8b39b-0bd4-4167-8f94-523be848cbde (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:01:29,238 [246] [DEBUG] [app] Starting request: urn:request:c842a402-81ad-4163-a39f-d1279203946c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:29,238 [245] [DEBUG] [app] Starting request: urn:request:68e10d47-1c15-426d-9112-df66eedf04cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:29,238 [246] [DEBUG] [app] Ending request: urn:request:c842a402-81ad-4163-a39f-d1279203946c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c842a402-81ad-4163-a39f-d1279203946c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:01:29,239 [245] [DEBUG] [app] Ending request: urn:request:68e10d47-1c15-426d-9112-df66eedf04cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68e10d47-1c15-426d-9112-df66eedf04cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:01:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:01:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:29,241 [238] [DEBUG] [app] Starting request: urn:request:389e8d2d-1d37-438c-8d47-94eeb9f1290b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:29,241 [236] [DEBUG] [app] Starting request: urn:request:0f97cf39-cb15-4d38-97b5-e14b30b53faa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:29,242 [238] [DEBUG] [app] Ending request: urn:request:389e8d2d-1d37-438c-8d47-94eeb9f1290b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:389e8d2d-1d37-438c-8d47-94eeb9f1290b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:01:29,242 [236] [DEBUG] [app] Ending request: urn:request:0f97cf39-cb15-4d38-97b5-e14b30b53faa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0f97cf39-cb15-4d38-97b5-e14b30b53faa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:01:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:01:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:29,254 [237] [DEBUG] [app] Ending request: urn:request:27047516-6fa6-4955-9c41-1c780778adfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:27047516-6fa6-4955-9c41-1c780778adfe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:29,254 [235] [DEBUG] [app] Ending request: urn:request:97c8b39b-0bd4-4167-8f94-523be848cbde (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97c8b39b-0bd4-4167-8f94-523be848cbde', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:01:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:01:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:01:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
queuecleanupworker stdout | 2025-02-07 04:01:29,954 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:01:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:01:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:32.625945+00:00 (in 1.004364 seconds)
namespacegcworker stdout | 2025-02-07 04:01:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:31 UTC)" (scheduled at 2025-02-07 04:01:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:01:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:01:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 31, 621896), True, datetime.datetime(2025, 2, 7, 4, 1, 31, 621896), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:01:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:01:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:01:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:01:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:01:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:13.624301+00:00 (in 40.997900 seconds)
namespacegcworker stdout | 2025-02-07 04:01:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:32 UTC)" (scheduled at 2025-02-07 04:01:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:01:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:01:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:01:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:01:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:03.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 04:01:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:03 UTC)" (scheduled at 2025-02-07 04:01:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:01:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 1, 33, 920003), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:01:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:01:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:01:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:01:34,244 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:01:35,218 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:56.713547+00:00 (in 18.997181 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:37 UTC)" (scheduled at 2025-02-07 04:01:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:01:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:01:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:48.732147+00:00 (in 9.999425 seconds)
notificationworker stdout | 2025-02-07 04:01:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:48 UTC)" (scheduled at 2025-02-07 04:01:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:01:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:01:38,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 38, 733165), True, datetime.datetime(2025, 2, 7, 4, 1, 38, 733165), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:01:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:01:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:01:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:01:39,320 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:01:39,834 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:01:40,402 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 04:01:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:01:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:54.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 04:01:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:10 UTC)" (scheduled at 2025-02-07 04:01:40.424250+00:00)
gcworker stdout | 2025-02-07 04:01:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:01:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900600431, None, 1, 0])
gcworker stdout | 2025-02-07 04:01:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:01:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:10 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:01:41,597 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:01:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:01:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:11.831521+00:00 (in 29.999527 seconds)
autopruneworker stdout | 2025-02-07 04:01:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:11 UTC)" (scheduled at 2025-02-07 04:01:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:01:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897301836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:01:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:01:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:01:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:01:42,509 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:01:42,622 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:01:43,677 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:43,994 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:01:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:01:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:59.225378+00:00 (in 14.999555 seconds)
quotatotalworker stdout | 2025-02-07 04:01:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:59 UTC)" (scheduled at 2025-02-07 04:01:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:01:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:01:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:01:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:01:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:01:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:01:44,236 [235] [DEBUG] [app] Starting request: urn:request:e31b0b5a-c2cb-4166-942e-488ca3419cfe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:44,236 [237] [DEBUG] [app] Starting request: urn:request:4f2d216d-16ca-40ed-b912-327c3c74aaac (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:01:44,239 [239] [DEBUG] [app] Starting request: urn:request:91dc0a89-d16a-4cfe-b10d-79c5f63515e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:44,239 [246] [DEBUG] [app] Starting request: urn:request:a65b1787-0ff6-41bd-8758-86d869e4ad5b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:44,239 [239] [DEBUG] [app] Ending request: urn:request:91dc0a89-d16a-4cfe-b10d-79c5f63515e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:91dc0a89-d16a-4cfe-b10d-79c5f63515e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:01:44,239 [246] [DEBUG] [app] Ending request: urn:request:a65b1787-0ff6-41bd-8758-86d869e4ad5b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a65b1787-0ff6-41bd-8758-86d869e4ad5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:01:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:01:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:01:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:44,243 [237] [DEBUG] [app] Starting request: urn:request:d11d7cb4-a2b0-47a0-a2a3-1fd4d678988c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:44,243 [236] [DEBUG] [app] Starting request: urn:request:ba8e4d4d-4ceb-44d2-8d53-f7516139abc1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:44,243 [237] [DEBUG] [app] Ending request: urn:request:d11d7cb4-a2b0-47a0-a2a3-1fd4d678988c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d11d7cb4-a2b0-47a0-a2a3-1fd4d678988c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:01:44,243 [236] [DEBUG] [app] Ending request: urn:request:ba8e4d4d-4ceb-44d2-8d53-f7516139abc1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba8e4d4d-4ceb-44d2-8d53-f7516139abc1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:01:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:01:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:44,255 [235] [DEBUG] [app] Ending request: urn:request:e31b0b5a-c2cb-4166-942e-488ca3419cfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e31b0b5a-c2cb-4166-942e-488ca3419cfe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:44,255 [237] [DEBUG] [app] Ending request: urn:request:4f2d216d-16ca-40ed-b912-327c3c74aaac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f2d216d-16ca-40ed-b912-327c3c74aaac', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:01:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:01:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:01:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
servicekey stdout | 2025-02-07 04:01:44,711 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:01:44,734 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:01:44,835 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:01:45,216 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:01:45,579 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:01:45,769 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:01:45,781 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:01:46,298 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:01:46,300 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:01:46,303 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:01:46,306 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:01:46,310 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:01:47,332 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:01:47,335 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:01:47,337 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:01:47,393 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:01:47,642 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:01:47,890 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:01:48,389 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:01:48,396 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:01:48,401 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:01:48,404 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:01:48,407 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:01:48,410 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:01:48,414 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:01:48,417 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:01:48,419 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:01:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:01:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:50.814677+00:00 (in 2.082036 seconds)
notificationworker stdout | 2025-02-07 04:01:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:58 UTC)" (scheduled at 2025-02-07 04:01:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:01:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:01:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 48, 732971), True, datetime.datetime(2025, 2, 7, 4, 1, 48, 732971), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:01:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:01:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:01:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:01:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:01:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:01:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:01:58.732147+00:00 (in 7.917031 seconds)
notificationworker stdout | 2025-02-07 04:01:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:50 UTC)" (scheduled at 2025-02-07 04:01:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:01:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:01:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:01:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:01:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:10.424250+00:00 (in 16.000702 seconds)
gcworker stdout | 2025-02-07 04:01:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:24 UTC)" (scheduled at 2025-02-07 04:01:54.423055+00:00)
gcworker stdout | 2025-02-07 04:01:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:01:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:01:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:01:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:56.118855+00:00 (in 59.999476 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:01:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:56 UTC)" (scheduled at 2025-02-07 04:01:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:01:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:01:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:01:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:37.715878+00:00 (in 41.001874 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:56 UTC)" (scheduled at 2025-02-07 04:01:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:01:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:01:56,907 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:01:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:01:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:08.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:01:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:08 UTC)" (scheduled at 2025-02-07 04:01:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:01:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:01:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 1, 58, 732904), True, datetime.datetime(2025, 2, 7, 4, 1, 58, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:01:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:01:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:01:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:01:58,800 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:01:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:01:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:14.225378+00:00 (in 14.999543 seconds)
quotatotalworker stdout | 2025-02-07 04:01:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:14 UTC)" (scheduled at 2025-02-07 04:01:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:01:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:01:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:01:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:01:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:01:59,235 [238] [DEBUG] [app] Starting request: urn:request:0d1f9796-1a5e-40fb-9acd-0473ac9c328b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:59,235 [235] [DEBUG] [app] Starting request: urn:request:bd8b1135-1290-483a-830d-7211012b463b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:01:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:01:59,238 [239] [DEBUG] [app] Starting request: urn:request:1a829fbc-441f-43d5-9a95-8d1ec4aebba7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:59,238 [244] [DEBUG] [app] Starting request: urn:request:28e778ff-9299-4518-9a6f-ff2f9b6153fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:01:59,238 [239] [DEBUG] [app] Ending request: urn:request:1a829fbc-441f-43d5-9a95-8d1ec4aebba7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1a829fbc-441f-43d5-9a95-8d1ec4aebba7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:01:59,238 [244] [DEBUG] [app] Ending request: urn:request:28e778ff-9299-4518-9a6f-ff2f9b6153fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28e778ff-9299-4518-9a6f-ff2f9b6153fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:01:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:01:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:01:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:01:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:01:59,241 [235] [DEBUG] [app] Starting request: urn:request:3d680946-2970-4cb9-b3d3-b6670d4c6007 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:59,241 [235] [DEBUG] [app] Ending request: urn:request:3d680946-2970-4cb9-b3d3-b6670d4c6007 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d680946-2970-4cb9-b3d3-b6670d4c6007', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:01:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:01:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:59,242 [236] [DEBUG] [app] Starting request: urn:request:9d03f002-ad04-4b1c-ae0a-56b7c325903b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:01:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:59,242 [236] [DEBUG] [app] Ending request: urn:request:9d03f002-ad04-4b1c-ae0a-56b7c325903b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9d03f002-ad04-4b1c-ae0a-56b7c325903b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:01:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:01:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:01:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:01:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:01:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:01:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:01:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:01:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:01:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:01:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:01:59,253 [235] [DEBUG] [app] Ending request: urn:request:bd8b1135-1290-483a-830d-7211012b463b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd8b1135-1290-483a-830d-7211012b463b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 04:01:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:01:59,254 [238] [DEBUG] [app] Ending request: urn:request:0d1f9796-1a5e-40fb-9acd-0473ac9c328b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d1f9796-1a5e-40fb-9acd-0473ac9c328b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:01:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:01:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 04:01:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:01:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:01:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:01:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:04.936688+00:00 (in 4.998273 seconds)
exportactionlogsworker stdout | 2025-02-07 04:01:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:59 UTC)" (scheduled at 2025-02-07 04:01:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:01:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:01:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:02:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:01:59,990 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:02:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:02:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:23.420886+00:00 (in 20.001323 seconds)
repositorygcworker stdout | 2025-02-07 04:02:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:03 UTC)" (scheduled at 2025-02-07 04:02:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:02:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:02:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 3, 419832), True, datetime.datetime(2025, 2, 7, 4, 2, 3, 419832), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:02:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:02:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:02:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:02:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:02:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:33.919286+00:00 (in 29.999531 seconds)
buildlogsarchiver stdout | 2025-02-07 04:02:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:33 UTC)" (scheduled at 2025-02-07 04:02:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:02:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 2, 3, 920026), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:02:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:02:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:02:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:02:04,274 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:02:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:02:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:59.937899+00:00 (in 55.000765 seconds)
exportactionlogsworker stdout | 2025-02-07 04:02:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:04 UTC)" (scheduled at 2025-02-07 04:02:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:02:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:02:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 4, 937472), True, datetime.datetime(2025, 2, 7, 4, 2, 4, 937472), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:02:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:02:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:02:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:02:05,254 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:02:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:18.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 04:02:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:18 UTC)" (scheduled at 2025-02-07 04:02:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:02:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:02:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 8, 732863), True, datetime.datetime(2025, 2, 7, 4, 2, 8, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:02:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:02:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:02:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:02:09,356 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:02:09,860 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:02:10,413 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gcworker stdout | 2025-02-07 04:02:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:02:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:24.423055+00:00 (in 13.998398 seconds)
gcworker stdout | 2025-02-07 04:02:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:40 UTC)" (scheduled at 2025-02-07 04:02:10.424250+00:00)
gcworker stdout | 2025-02-07 04:02:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:02:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900630430, None, 1, 0])
gcworker stdout | 2025-02-07 04:02:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:02:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:40 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:02:11,634 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:02:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:02:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:41.831521+00:00 (in 29.999539 seconds)
autopruneworker stdout | 2025-02-07 04:02:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:41 UTC)" (scheduled at 2025-02-07 04:02:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:02:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897331836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:02:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:02:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:02:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:02:12,546 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:02:12,658 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
namespacegcworker stdout | 2025-02-07 04:02:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:02:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:31.621131+00:00 (in 17.996369 seconds)
namespacegcworker stdout | 2025-02-07 04:02:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:07:13 UTC)" (scheduled at 2025-02-07 04:02:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:02:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 2, 13, 625024), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:02:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 2, 13, 625024), True, datetime.datetime(2025, 2, 7, 4, 2, 13, 625024), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:02:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 2, 13, 625024), True, datetime.datetime(2025, 2, 7, 4, 2, 13, 625024), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 2, 13, 625024), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:02:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:02:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:07:13 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:02:13,713 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:14,030 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:02:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:02:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:29.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 04:02:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:29 UTC)" (scheduled at 2025-02-07 04:02:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:02:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:02:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:02:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:02:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:02:14,235 [235] [DEBUG] [app] Starting request: urn:request:73e7b29f-a15c-46b6-9c19-4751db682f52 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:14,235 [236] [DEBUG] [app] Starting request: urn:request:d7736f67-4de8-4a68-a48d-f321a8da3c6b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:02:14,238 [246] [DEBUG] [app] Starting request: urn:request:2812c026-8501-4fce-bcd9-b107760e1f7d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:14,239 [245] [DEBUG] [app] Starting request: urn:request:de523536-590e-4660-ae40-2589ceb9d804 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:14,239 [246] [DEBUG] [app] Ending request: urn:request:2812c026-8501-4fce-bcd9-b107760e1f7d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2812c026-8501-4fce-bcd9-b107760e1f7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:02:14,239 [245] [DEBUG] [app] Ending request: urn:request:de523536-590e-4660-ae40-2589ceb9d804 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:de523536-590e-4660-ae40-2589ceb9d804', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:02:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:02:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:14,242 [235] [DEBUG] [app] Starting request: urn:request:7baa602a-52ea-409f-9931-66e45c932619 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:14,242 [235] [DEBUG] [app] Ending request: urn:request:7baa602a-52ea-409f-9931-66e45c932619 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7baa602a-52ea-409f-9931-66e45c932619', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:02:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:14,243 [238] [DEBUG] [app] Starting request: urn:request:4bc111c5-2689-416b-b02d-77834f685a6e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:14,243 [238] [DEBUG] [app] Ending request: urn:request:4bc111c5-2689-416b-b02d-77834f685a6e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4bc111c5-2689-416b-b02d-77834f685a6e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:02:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:02:14,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:14,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:14,254 [235] [DEBUG] [app] Ending request: urn:request:73e7b29f-a15c-46b6-9c19-4751db682f52 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:73e7b29f-a15c-46b6-9c19-4751db682f52', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:02:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:02:14,255 [236] [DEBUG] [app] Ending request: urn:request:d7736f67-4de8-4a68-a48d-f321a8da3c6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d7736f67-4de8-4a68-a48d-f321a8da3c6b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:02:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
servicekey stdout | 2025-02-07 04:02:14,748 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:02:14,750 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:02:14,871 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:02:15,228 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:02:15,606 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:02:15,805 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:02:15,816 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:02:16,306 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:02:16,309 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:02:16,311 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:02:16,315 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:02:16,318 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:02:17,340 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:02:17,343 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:02:17,346 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:02:17,429 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:02:17,670 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:02:17,901 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:02:18,403 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:02:18,407 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:02:18,411 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:02:18,416 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:02:18,418 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:02:18,422 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:02:18,424 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:02:18,427 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:02:18,429 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:02:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:28.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:02:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:28 UTC)" (scheduled at 2025-02-07 04:02:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:02:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:02:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 18, 732864), True, datetime.datetime(2025, 2, 7, 4, 2, 18, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:02:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:02:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:02:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:02:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:02:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:03.419110+00:00 (in 39.997749 seconds)
repositorygcworker stdout | 2025-02-07 04:02:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:23 UTC)" (scheduled at 2025-02-07 04:02:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:02:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:02:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:02:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:02:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:40.424250+00:00 (in 16.000735 seconds)
gcworker stdout | 2025-02-07 04:02:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:54 UTC)" (scheduled at 2025-02-07 04:02:24.423055+00:00)
gcworker stdout | 2025-02-07 04:02:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:02:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:02:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:02:26,935 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:02:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:33.814166+00:00 (in 5.081556 seconds)
notificationworker stdout | 2025-02-07 04:02:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:38 UTC)" (scheduled at 2025-02-07 04:02:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:02:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:02:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 28, 732925), True, datetime.datetime(2025, 2, 7, 4, 2, 28, 732925), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:02:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:02:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:02:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:02:28,836 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:02:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:02:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:44.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:02:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:44 UTC)" (scheduled at 2025-02-07 04:02:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:02:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:02:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:02:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:02:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:02:29,235 [237] [DEBUG] [app] Starting request: urn:request:d83ae035-14d7-49e3-bd93-cd48285d59fb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:29,235 [235] [DEBUG] [app] Starting request: urn:request:d8917fc1-3b6e-4cc0-a226-db188c287cf3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:02:29,238 [246] [DEBUG] [app] Starting request: urn:request:5468f49c-618d-480c-9943-2ac1b855ab20 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:29,238 [245] [DEBUG] [app] Starting request: urn:request:7692e03c-b397-4197-a806-160e3f624a4f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:29,238 [245] [DEBUG] [app] Ending request: urn:request:7692e03c-b397-4197-a806-160e3f624a4f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7692e03c-b397-4197-a806-160e3f624a4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:02:29,238 [246] [DEBUG] [app] Ending request: urn:request:5468f49c-618d-480c-9943-2ac1b855ab20 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5468f49c-618d-480c-9943-2ac1b855ab20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:02:29,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:02:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:02:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:29,241 [235] [DEBUG] [app] Starting request: urn:request:49582831-d2c8-4bce-a318-688a02d43aed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:29,241 [237] [DEBUG] [app] Starting request: urn:request:87c3e621-391a-4890-a3af-4d2b6633b9c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:29,241 [235] [DEBUG] [app] Ending request: urn:request:49582831-d2c8-4bce-a318-688a02d43aed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:49582831-d2c8-4bce-a318-688a02d43aed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:02:29,241 [237] [DEBUG] [app] Ending request: urn:request:87c3e621-391a-4890-a3af-4d2b6633b9c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87c3e621-391a-4890-a3af-4d2b6633b9c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:02:29,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:29,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:29,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:29,245 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:29,245 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:29,253 [235] [DEBUG] [app] Ending request: urn:request:d8917fc1-3b6e-4cc0-a226-db188c287cf3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d8917fc1-3b6e-4cc0-a226-db188c287cf3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:02:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:02:29,254 [237] [DEBUG] [app] Ending request: urn:request:d83ae035-14d7-49e3-bd93-cd48285d59fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d83ae035-14d7-49e3-bd93-cd48285d59fb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:02:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 04:02:30,010 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:02:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:02:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:32.625945+00:00 (in 1.004316 seconds)
namespacegcworker stdout | 2025-02-07 04:02:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:31 UTC)" (scheduled at 2025-02-07 04:02:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:02:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:02:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 31, 621914), True, datetime.datetime(2025, 2, 7, 4, 2, 31, 621914), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:02:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:02:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:02:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:02:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:02:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:31.621131+00:00 (in 58.994727 seconds)
namespacegcworker stdout | 2025-02-07 04:02:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:32 UTC)" (scheduled at 2025-02-07 04:02:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:02:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:02:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:02:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:38.732147+00:00 (in 4.917489 seconds)
notificationworker stdout | 2025-02-07 04:02:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:07:33 UTC)" (scheduled at 2025-02-07 04:02:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:02:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 2, 33, 814916), 'notification/%'])
notificationworker stdout | 2025-02-07 04:02:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 2, 33, 814916), True, datetime.datetime(2025, 2, 7, 4, 2, 33, 814916), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:02:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 2, 33, 814916), True, datetime.datetime(2025, 2, 7, 4, 2, 33, 814916), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 2, 33, 814916), 'notification/%'])
notificationworker stdout | 2025-02-07 04:02:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:02:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:07:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:02:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:02:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:03.919286+00:00 (in 29.999559 seconds)
buildlogsarchiver stdout | 2025-02-07 04:02:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:03 UTC)" (scheduled at 2025-02-07 04:02:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:02:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 2, 33, 919971), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:02:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:02:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:02:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:02:34,293 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:02:35,291 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:56.713547+00:00 (in 18.997177 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:37 UTC)" (scheduled at 2025-02-07 04:02:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:02:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:48.732147+00:00 (in 9.999507 seconds)
notificationworker stdout | 2025-02-07 04:02:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:48 UTC)" (scheduled at 2025-02-07 04:02:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:02:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:02:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 38, 732967), True, datetime.datetime(2025, 2, 7, 4, 2, 38, 732967), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:02:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:02:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:02:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:02:39,386 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:02:39,875 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:02:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:02:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:54.423055+00:00 (in 13.998356 seconds)
gcworker stdout | 2025-02-07 04:02:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:10 UTC)" (scheduled at 2025-02-07 04:02:40.424250+00:00)
gcworker stdout | 2025-02-07 04:02:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:02:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900660431, None, 1, 0])
gcworker stdout | 2025-02-07 04:02:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:02:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:02:40,436 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:02:41,671 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:02:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:02:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:11.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 04:02:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:11 UTC)" (scheduled at 2025-02-07 04:02:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:02:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897361836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:02:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:02:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:02:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:02:42,578 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:02:42,694 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:02:43,732 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:44,061 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:02:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:02:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:59.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 04:02:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:59 UTC)" (scheduled at 2025-02-07 04:02:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:02:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:02:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:02:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:02:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:02:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:02:44,235 [237] [DEBUG] [app] Starting request: urn:request:c4e20ab4-8eeb-4f8b-bc09-3bfd165136be (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:44,235 [238] [DEBUG] [app] Starting request: urn:request:b5118f85-3bf9-4dba-a116-2f9dbb0471e7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:02:44,238 [239] [DEBUG] [app] Starting request: urn:request:4ffa8a27-3ba4-4856-a30c-668e4bf6fbb5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:44,238 [246] [DEBUG] [app] Starting request: urn:request:df259b59-5140-4901-bd1a-c7049fb23034 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:44,239 [239] [DEBUG] [app] Ending request: urn:request:4ffa8a27-3ba4-4856-a30c-668e4bf6fbb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ffa8a27-3ba4-4856-a30c-668e4bf6fbb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:02:44,239 [246] [DEBUG] [app] Ending request: urn:request:df259b59-5140-4901-bd1a-c7049fb23034 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df259b59-5140-4901-bd1a-c7049fb23034', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:02:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:02:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:02:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:44,241 [236] [DEBUG] [app] Starting request: urn:request:046863a9-31cb-438d-a893-5a5ce5d6a91b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:44,242 [236] [DEBUG] [app] Ending request: urn:request:046863a9-31cb-438d-a893-5a5ce5d6a91b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:046863a9-31cb-438d-a893-5a5ce5d6a91b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:02:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:44,243 [236] [DEBUG] [app] Starting request: urn:request:fdc8841c-ef21-48c1-8a61-fe603fc71680 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:44,243 [236] [DEBUG] [app] Ending request: urn:request:fdc8841c-ef21-48c1-8a61-fe603fc71680 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdc8841c-ef21-48c1-8a61-fe603fc71680', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:02:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:02:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:44,253 [237] [DEBUG] [app] Ending request: urn:request:c4e20ab4-8eeb-4f8b-bc09-3bfd165136be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4e20ab4-8eeb-4f8b-bc09-3bfd165136be', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:02:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:02:44,254 [238] [DEBUG] [app] Ending request: urn:request:b5118f85-3bf9-4dba-a116-2f9dbb0471e7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5118f85-3bf9-4dba-a116-2f9dbb0471e7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:02:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
globalpromstats stdout | 2025-02-07 04:02:44,758 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:02:44,783 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:02:44,901 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:02:45,264 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:02:45,642 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:02:45,835 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:02:45,846 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:02:46,314 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:02:46,318 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:02:46,323 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:02:46,326 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:02:46,329 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:02:47,348 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:02:47,351 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:02:47,353 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:02:47,466 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:02:47,690 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:02:47,937 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:02:48,415 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:02:48,418 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:02:48,421 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:02:48,424 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:02:48,427 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:02:48,430 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:02:48,433 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:02:48,436 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:02:48,439 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:02:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:50.814677+00:00 (in 2.082029 seconds)
notificationworker stdout | 2025-02-07 04:02:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:58 UTC)" (scheduled at 2025-02-07 04:02:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:02:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:02:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 48, 732870), True, datetime.datetime(2025, 2, 7, 4, 2, 48, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:02:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:02:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:02:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:02:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:02:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:02:58.732147+00:00 (in 7.916946 seconds)
notificationworker stdout | 2025-02-07 04:02:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:50 UTC)" (scheduled at 2025-02-07 04:02:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:02:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:02:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:02:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:02:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:10.424250+00:00 (in 16.000751 seconds)
gcworker stdout | 2025-02-07 04:02:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:24 UTC)" (scheduled at 2025-02-07 04:02:54.423055+00:00)
gcworker stdout | 2025-02-07 04:02:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:02:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:02:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:02:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:56.118855+00:00 (in 59.999518 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:02:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:56 UTC)" (scheduled at 2025-02-07 04:02:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:02:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:02:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:02:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:37.715878+00:00 (in 41.001895 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:56 UTC)" (scheduled at 2025-02-07 04:02:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:02:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:02:56,961 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:02:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:02:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:08.732147+00:00 (in 9.999442 seconds)
notificationworker stdout | 2025-02-07 04:02:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:08 UTC)" (scheduled at 2025-02-07 04:02:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:02:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:02:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 2, 58, 733023), True, datetime.datetime(2025, 2, 7, 4, 2, 58, 733023), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:02:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:02:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:02:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:02:58,853 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:02:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:02:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:14.225378+00:00 (in 14.999604 seconds)
quotatotalworker stdout | 2025-02-07 04:02:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:14 UTC)" (scheduled at 2025-02-07 04:02:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:02:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:02:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:02:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:02:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:02:59,235 [235] [DEBUG] [app] Starting request: urn:request:36aace9a-c257-4129-b6ab-f72a07fb8a16 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:59,235 [236] [DEBUG] [app] Starting request: urn:request:41b371f2-b644-457a-aa22-f8e3cf233a6c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:02:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:02:59,238 [245] [DEBUG] [app] Starting request: urn:request:9e080d6d-918b-456e-80a4-b362257804d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:59,238 [244] [DEBUG] [app] Starting request: urn:request:951822b7-151b-4ec5-873b-485b9cb92ecc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:02:59,238 [244] [DEBUG] [app] Ending request: urn:request:951822b7-151b-4ec5-873b-485b9cb92ecc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:951822b7-151b-4ec5-873b-485b9cb92ecc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:02:59,238 [245] [DEBUG] [app] Ending request: urn:request:9e080d6d-918b-456e-80a4-b362257804d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e080d6d-918b-456e-80a4-b362257804d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:02:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:02:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:02:59,241 [236] [DEBUG] [app] Starting request: urn:request:5cf30f5d-09db-48a6-90ec-c5cee68dfb8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:59,242 [236] [DEBUG] [app] Ending request: urn:request:5cf30f5d-09db-48a6-90ec-c5cee68dfb8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5cf30f5d-09db-48a6-90ec-c5cee68dfb8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:02:59,242 [237] [DEBUG] [app] Starting request: urn:request:d1ba0691-5989-49c1-8df5-e127463b97ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:02:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:02:59,242 [237] [DEBUG] [app] Ending request: urn:request:d1ba0691-5989-49c1-8df5-e127463b97ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d1ba0691-5989-49c1-8df5-e127463b97ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:02:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:02:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:02:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:02:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:02:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:02:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:02:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:02:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:02:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:02:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:02:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:02:59,255 [235] [DEBUG] [app] Ending request: urn:request:36aace9a-c257-4129-b6ab-f72a07fb8a16 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:36aace9a-c257-4129-b6ab-f72a07fb8a16', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:59,255 [236] [DEBUG] [app] Ending request: urn:request:41b371f2-b644-457a-aa22-f8e3cf233a6c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:41b371f2-b644-457a-aa22-f8e3cf233a6c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:02:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:02:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:02:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:02:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:02:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:02:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:02:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:02:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:04.936688+00:00 (in 4.998308 seconds)
exportactionlogsworker stdout | 2025-02-07 04:02:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:59 UTC)" (scheduled at 2025-02-07 04:02:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:02:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:02:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:03:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:03:00,046 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:03:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:03:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:23.420886+00:00 (in 20.001302 seconds)
repositorygcworker stdout | 2025-02-07 04:03:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:03 UTC)" (scheduled at 2025-02-07 04:03:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:03:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:03:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 3, 419879), True, datetime.datetime(2025, 2, 7, 4, 3, 3, 419879), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:03:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:03:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:03:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:03:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:03:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:33.919286+00:00 (in 29.999526 seconds)
buildlogsarchiver stdout | 2025-02-07 04:03:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:33 UTC)" (scheduled at 2025-02-07 04:03:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:03:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 3, 3, 920045), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:03:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:03:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:03:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:03:04,318 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:03:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:03:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:59.937899+00:00 (in 55.000779 seconds)
exportactionlogsworker stdout | 2025-02-07 04:03:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:04 UTC)" (scheduled at 2025-02-07 04:03:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:03:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:03:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 4, 937422), True, datetime.datetime(2025, 2, 7, 4, 3, 4, 937422), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:03:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:03:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:03:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:03:05,314 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:03:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:03:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:18.732147+00:00 (in 9.999482 seconds)
notificationworker stdout | 2025-02-07 04:03:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:18 UTC)" (scheduled at 2025-02-07 04:03:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:03:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:03:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 8, 732974), True, datetime.datetime(2025, 2, 7, 4, 3, 8, 732974), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:03:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:03:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:03:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:03:09,411 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:03:09,911 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:03:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:03:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:24.423055+00:00 (in 13.998384 seconds)
gcworker stdout | 2025-02-07 04:03:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:40 UTC)" (scheduled at 2025-02-07 04:03:10.424250+00:00)
gcworker stdout | 2025-02-07 04:03:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:03:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900690431, None, 1, 0])
gcworker stdout | 2025-02-07 04:03:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:03:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:03:10,450 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:03:11,707 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:03:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:03:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:41.831521+00:00 (in 29.999540 seconds)
autopruneworker stdout | 2025-02-07 04:03:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:41 UTC)" (scheduled at 2025-02-07 04:03:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:03:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897391836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:03:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:03:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:03:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:03:12,588 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:03:12,711 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:03:13,769 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:14,067 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:03:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:03:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:29.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 04:03:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:29 UTC)" (scheduled at 2025-02-07 04:03:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:03:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:03:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:03:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:03:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:03:14,235 [235] [DEBUG] [app] Starting request: urn:request:394e7db7-a31a-418d-84fb-73490e1f65fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:14,235 [237] [DEBUG] [app] Starting request: urn:request:8cf0e4dd-8494-4264-bb5e-5ddc1d7bde0e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:03:14,238 [239] [DEBUG] [app] Starting request: urn:request:04ac8a11-262e-4265-a0b1-7d64f7582fe4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:14,238 [246] [DEBUG] [app] Starting request: urn:request:6adb7cd6-63a0-4449-ba94-c1ecad5d9cc0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:14,238 [246] [DEBUG] [app] Ending request: urn:request:6adb7cd6-63a0-4449-ba94-c1ecad5d9cc0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6adb7cd6-63a0-4449-ba94-c1ecad5d9cc0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:03:14,238 [239] [DEBUG] [app] Ending request: urn:request:04ac8a11-262e-4265-a0b1-7d64f7582fe4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04ac8a11-262e-4265-a0b1-7d64f7582fe4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:03:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:03:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:14,241 [236] [DEBUG] [app] Starting request: urn:request:16cdf9e5-5fe4-496e-a036-03e31b221f37 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:14,242 [236] [DEBUG] [app] Ending request: urn:request:16cdf9e5-5fe4-496e-a036-03e31b221f37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16cdf9e5-5fe4-496e-a036-03e31b221f37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:03:14,242 [235] [DEBUG] [app] Starting request: urn:request:318002b0-1b06-4b99-825c-7e30b22840ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:03:14,242 [235] [DEBUG] [app] Ending request: urn:request:318002b0-1b06-4b99-825c-7e30b22840ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:318002b0-1b06-4b99-825c-7e30b22840ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:03:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:03:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:03:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:03:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:14,253 [235] [DEBUG] [app] Ending request: urn:request:394e7db7-a31a-418d-84fb-73490e1f65fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:394e7db7-a31a-418d-84fb-73490e1f65fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:14,254 [237] [DEBUG] [app] Ending request: urn:request:8cf0e4dd-8494-4264-bb5e-5ddc1d7bde0e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8cf0e4dd-8494-4264-bb5e-5ddc1d7bde0e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:03:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
globalpromstats stdout | 2025-02-07 04:03:14,786 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:03:14,804 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:03:14,917 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:03:15,300 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:03:15,679 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:03:15,854 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:03:15,875 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:03:16,323 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:03:16,325 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:03:16,330 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:03:16,333 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:03:16,335 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:03:17,355 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:03:17,358 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:03:17,361 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:03:17,502 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:03:17,724 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:03:17,948 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:03:18,427 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:03:18,430 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:03:18,433 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:03:18,437 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:03:18,439 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:03:18,442 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:03:18,444 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:03:18,447 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:03:18,449 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:03:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:03:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:28.732147+00:00 (in 9.999521 seconds)
notificationworker stdout | 2025-02-07 04:03:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:28 UTC)" (scheduled at 2025-02-07 04:03:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:03:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:03:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 18, 732905), True, datetime.datetime(2025, 2, 7, 4, 3, 18, 732905), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:03:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:03:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:03:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:03:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:03:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:32.420707+00:00 (in 8.999350 seconds)
repositorygcworker stdout | 2025-02-07 04:03:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:23 UTC)" (scheduled at 2025-02-07 04:03:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:03:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:03:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:03:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:03:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:40.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 04:03:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:54 UTC)" (scheduled at 2025-02-07 04:03:24.423055+00:00)
gcworker stdout | 2025-02-07 04:03:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:03:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:03:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:03:26,970 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:03:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:03:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:38.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:03:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:38 UTC)" (scheduled at 2025-02-07 04:03:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:03:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:03:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 28, 732858), True, datetime.datetime(2025, 2, 7, 4, 3, 28, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:03:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:03:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:03:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:03:28,890 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:03:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:03:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:44.225378+00:00 (in 14.999561 seconds)
quotatotalworker stdout | 2025-02-07 04:03:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:44 UTC)" (scheduled at 2025-02-07 04:03:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:03:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:03:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:03:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:03:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:03:29,235 [238] [DEBUG] [app] Starting request: urn:request:f7111180-537c-48ce-8e96-1eec2b7a3eb3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:29,235 [236] [DEBUG] [app] Starting request: urn:request:9728f8a9-2fb5-43c2-8264-29cd45f8c456 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:03:29,238 [246] [DEBUG] [app] Starting request: urn:request:8bcc4bdd-69cb-466e-a3dd-5be71db8a0cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:29,238 [239] [DEBUG] [app] Starting request: urn:request:fc054f95-0b8a-463c-8afd-ee4204f6aa6e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:29,239 [239] [DEBUG] [app] Ending request: urn:request:fc054f95-0b8a-463c-8afd-ee4204f6aa6e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc054f95-0b8a-463c-8afd-ee4204f6aa6e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:03:29,239 [246] [DEBUG] [app] Ending request: urn:request:8bcc4bdd-69cb-466e-a3dd-5be71db8a0cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8bcc4bdd-69cb-466e-a3dd-5be71db8a0cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:03:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:03:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:03:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:29,242 [235] [DEBUG] [app] Starting request: urn:request:4eab9978-0ce9-4e3b-8f80-b286d8193129 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:29,242 [237] [DEBUG] [app] Starting request: urn:request:5f506253-6470-4d90-9d11-220308a932d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:29,242 [235] [DEBUG] [app] Ending request: urn:request:4eab9978-0ce9-4e3b-8f80-b286d8193129 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4eab9978-0ce9-4e3b-8f80-b286d8193129', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:03:29,243 [237] [DEBUG] [app] Ending request: urn:request:5f506253-6470-4d90-9d11-220308a932d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f506253-6470-4d90-9d11-220308a932d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:03:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:03:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:03:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:29,254 [238] [DEBUG] [app] Ending request: urn:request:f7111180-537c-48ce-8e96-1eec2b7a3eb3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f7111180-537c-48ce-8e96-1eec2b7a3eb3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:29,254 [236] [DEBUG] [app] Ending request: urn:request:9728f8a9-2fb5-43c2-8264-29cd45f8c456 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9728f8a9-2fb5-43c2-8264-29cd45f8c456', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:03:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
queuecleanupworker stdout | 2025-02-07 04:03:30,078 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:03:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:03:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:32.625945+00:00 (in 1.004307 seconds)
namespacegcworker stdout | 2025-02-07 04:03:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:31 UTC)" (scheduled at 2025-02-07 04:03:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:03:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:03:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 31, 621954), True, datetime.datetime(2025, 2, 7, 4, 3, 31, 621954), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:03:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:03:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:03:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:03:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:03:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:03.419110+00:00 (in 30.997954 seconds)
repositorygcworker stdout | 2025-02-07 04:03:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:08:32 UTC)" (scheduled at 2025-02-07 04:03:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:03:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 3, 32, 421370), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:03:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 3, 32, 421370), True, datetime.datetime(2025, 2, 7, 4, 3, 32, 421370), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:03:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 3, 32, 421370), True, datetime.datetime(2025, 2, 7, 4, 3, 32, 421370), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 3, 32, 421370), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:03:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:03:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:08:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:03:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:03:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:31.621131+00:00 (in 58.994607 seconds)
namespacegcworker stdout | 2025-02-07 04:03:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:32 UTC)" (scheduled at 2025-02-07 04:03:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:03:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:03:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:03:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:03:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:03.919286+00:00 (in 29.999514 seconds)
buildlogsarchiver stdout | 2025-02-07 04:03:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:03 UTC)" (scheduled at 2025-02-07 04:03:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:03:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 3, 33, 920031), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:03:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:03:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:03:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:03:34,354 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:03:35,340 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:56.713547+00:00 (in 18.997163 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:37 UTC)" (scheduled at 2025-02-07 04:03:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:03:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:03:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:48.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 04:03:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:48 UTC)" (scheduled at 2025-02-07 04:03:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:03:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:03:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 38, 732812), True, datetime.datetime(2025, 2, 7, 4, 3, 38, 732812), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:03:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:03:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:03:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:03:39,432 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:03:39,947 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:03:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:03:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:54.423055+00:00 (in 13.998376 seconds)
gcworker stdout | 2025-02-07 04:03:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:10 UTC)" (scheduled at 2025-02-07 04:03:40.424250+00:00)
gcworker stdout | 2025-02-07 04:03:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:03:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900720431, None, 1, 0])
gcworker stdout | 2025-02-07 04:03:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:03:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:03:40,466 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:03:41,743 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:03:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:03:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:11.831521+00:00 (in 29.999497 seconds)
autopruneworker stdout | 2025-02-07 04:03:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:11 UTC)" (scheduled at 2025-02-07 04:03:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:03:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897421836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:03:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:03:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:03:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:03:42,617 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:03:42,718 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:03:43,797 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:44,104 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:03:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:03:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:59.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 04:03:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:59 UTC)" (scheduled at 2025-02-07 04:03:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:03:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:03:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:03:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:03:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:03:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:03:44,235 [235] [DEBUG] [app] Starting request: urn:request:0f5e0f0f-5a65-454d-b939-bff73a96cf59 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:44,235 [236] [DEBUG] [app] Starting request: urn:request:236477af-9025-4d71-af0b-dd42dcf5dd8d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:03:44,238 [239] [DEBUG] [app] Starting request: urn:request:463f12ec-4ba5-41ea-970b-e5a6a7148ce3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:44,238 [246] [DEBUG] [app] Starting request: urn:request:67e971be-7fce-4b64-a755-44fea2095fb5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:44,239 [239] [DEBUG] [app] Ending request: urn:request:463f12ec-4ba5-41ea-970b-e5a6a7148ce3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:463f12ec-4ba5-41ea-970b-e5a6a7148ce3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:03:44,239 [246] [DEBUG] [app] Ending request: urn:request:67e971be-7fce-4b64-a755-44fea2095fb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:67e971be-7fce-4b64-a755-44fea2095fb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:03:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:03:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:44,243 [238] [DEBUG] [app] Starting request: urn:request:c6953d8e-3398-4c87-8aad-5b1bffb72393 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:44,243 [237] [DEBUG] [app] Starting request: urn:request:85222b99-5b5d-4d1a-86e2-d2279d256de8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:44,243 [238] [DEBUG] [app] Ending request: urn:request:c6953d8e-3398-4c87-8aad-5b1bffb72393 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6953d8e-3398-4c87-8aad-5b1bffb72393', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:03:44,243 [237] [DEBUG] [app] Ending request: urn:request:85222b99-5b5d-4d1a-86e2-d2279d256de8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:85222b99-5b5d-4d1a-86e2-d2279d256de8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:03:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:03:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:03:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:03:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:44,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:44,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:44,255 [235] [DEBUG] [app] Ending request: urn:request:0f5e0f0f-5a65-454d-b939-bff73a96cf59 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f5e0f0f-5a65-454d-b939-bff73a96cf59', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:44,256 [236] [DEBUG] [app] Ending request: urn:request:236477af-9025-4d71-af0b-dd42dcf5dd8d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:236477af-9025-4d71-af0b-dd42dcf5dd8d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:44,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:03:44,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
globalpromstats stdout | 2025-02-07 04:03:44,823 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:03:44,840 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:03:44,955 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:03:45,333 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:03:45,704 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:03:45,890 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:03:45,911 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:03:46,332 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:03:46,335 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:03:46,337 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:03:46,340 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:03:46,343 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:03:47,364 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:03:47,366 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:03:47,369 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:03:47,538 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:03:47,748 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:03:47,962 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:03:48,438 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:03:48,441 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:03:48,444 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:03:48,447 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:03:48,450 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:03:48,453 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:03:48,457 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:03:48,459 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:03:48,464 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:03:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:03:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:50.814677+00:00 (in 2.082077 seconds)
notificationworker stdout | 2025-02-07 04:03:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:58 UTC)" (scheduled at 2025-02-07 04:03:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:03:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:03:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 48, 732892), True, datetime.datetime(2025, 2, 7, 4, 3, 48, 732892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:03:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:03:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:03:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:03:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:03:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:03:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:03:58.732147+00:00 (in 7.917029 seconds)
notificationworker stdout | 2025-02-07 04:03:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:50 UTC)" (scheduled at 2025-02-07 04:03:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:03:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:03:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:03:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:03:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:10.424250+00:00 (in 16.000737 seconds)
gcworker stdout | 2025-02-07 04:03:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:24 UTC)" (scheduled at 2025-02-07 04:03:54.423055+00:00)
gcworker stdout | 2025-02-07 04:03:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:03:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:24 UTC)" executed successfully
servicekey stdout | 2025-02-07 04:03:55,326 [99] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2025-02-07 04:03:55,326 [99] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-02-07 04:03:55 UTC)" (scheduled at 2025-02-07 04:03:55.325482+00:00)
servicekey stdout | 2025-02-07 04:03:55,327 [99] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:55.325482+00:00 (in 3299.998222 seconds)
servicekey stdout | 2025-02-07 04:03:55,327 [99] [DEBUG] [__main__] Starting automatic refresh of service key rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4 to new expiration 2025-02-07 06:03:55.327128
servicekey stdout | 2025-02-07 04:03:55,328 [99] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4', datetime.datetime(2025, 1, 31, 4, 3, 55, 327655), None, 1, 0])
servicekey stdout | 2025-02-07 04:03:55,335 [99] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise-15131.apps.quaytest-15131.qe.devcluster.openshift.com:443', 'rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4', 'quay', '{"n": "ydZTwWkSP7GLo1VWTY2-tvY2zs-NGpukK9A_Pr1ftW_Qt6JvAWK7vQNC1EA6mULUpUnSEnza3ySALI1yWv9VMgurD8CDnre3MgeFTD3HgFNEnP5af62LG9xYf376yvB9fKjs0XIvd7kbOqdixWiBBR1AGkmOai4M3ehFLQ1vhqcdOlPSPvGgQsRwzTGY78K72BJvpsmB-7yGfu23ebGgLJPjKIwBoWFAsogIVyfxg3kF0x6hFdlu3vEnHpvlHXnAXZyZKfh8rKMbE30HRQGovuegeNsHmy7YpLgtZ-Khw8AKUMVEpkF5u6ljkF6SFN8UuEdD6Kz9s7O1xPBo2iKYqw", "e": "AQAB", "kty": "RSA", "kid": "rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4"}', '{"created_by": "CLI tool"}', datetime.datetime(2025, 2, 7, 3, 15, 32, 537814), datetime.datetime(2025, 2, 7, 6, 3, 55, 327128), None, 2, 2])
servicekey stdout | 2025-02-07 04:03:55,337 [99] [DEBUG] [__main__] Finished automatic refresh of service key rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4 with new expiration 2025-02-07 06:03:55.327128
servicekey stdout | 2025-02-07 04:03:55,337 [99] [DEBUG] [data.database] Disconnecting from database.
servicekey stdout | 2025-02-07 04:03:55,337 [99] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-02-07 04:58:55 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:03:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:03:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:56.118855+00:00 (in 59.999468 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:03:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:56 UTC)" (scheduled at 2025-02-07 04:03:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:03:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:03:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:03:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:37.715878+00:00 (in 41.001822 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:56 UTC)" (scheduled at 2025-02-07 04:03:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:03:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:03:56,987 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:03:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:03:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:08.732147+00:00 (in 9.999572 seconds)
notificationworker stdout | 2025-02-07 04:03:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:08 UTC)" (scheduled at 2025-02-07 04:03:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:03:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:03:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 3, 58, 732867), True, datetime.datetime(2025, 2, 7, 4, 3, 58, 732867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:03:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:03:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:03:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:03:58,926 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:03:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:03:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:14.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 04:03:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:14 UTC)" (scheduled at 2025-02-07 04:03:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:03:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:03:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:03:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:03:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:03:59,236 [237] [DEBUG] [app] Starting request: urn:request:e3f712ed-e8c4-4c70-8c35-a46b92092a47 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:59,236 [238] [DEBUG] [app] Starting request: urn:request:19d2c6cd-ba91-4798-9ebe-ded6d862fe10 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:03:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:03:59,238 [239] [DEBUG] [app] Starting request: urn:request:ead3579b-57f4-4625-9bbe-46cbbeacacd6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:59,239 [246] [DEBUG] [app] Starting request: urn:request:2f9e8152-0d46-4282-a02b-56d1503c207a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:03:59,239 [239] [DEBUG] [app] Ending request: urn:request:ead3579b-57f4-4625-9bbe-46cbbeacacd6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ead3579b-57f4-4625-9bbe-46cbbeacacd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:03:59,239 [246] [DEBUG] [app] Ending request: urn:request:2f9e8152-0d46-4282-a02b-56d1503c207a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f9e8152-0d46-4282-a02b-56d1503c207a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:03:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:03:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:03:59,242 [238] [DEBUG] [app] Starting request: urn:request:a8f9ba69-56a0-475e-bcf1-28c493ed57b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:59,242 [238] [DEBUG] [app] Ending request: urn:request:a8f9ba69-56a0-475e-bcf1-28c493ed57b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a8f9ba69-56a0-475e-bcf1-28c493ed57b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:03:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:59,242 [235] [DEBUG] [app] Starting request: urn:request:12cad702-e9cb-45d9-86d5-b65e097eaaf3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:03:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:59,243 [235] [DEBUG] [app] Ending request: urn:request:12cad702-e9cb-45d9-86d5-b65e097eaaf3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:12cad702-e9cb-45d9-86d5-b65e097eaaf3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:03:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:03:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:03:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:03:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:03:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:03:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:03:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:03:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:03:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:03:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:03:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:03:59,254 [238] [DEBUG] [app] Ending request: urn:request:19d2c6cd-ba91-4798-9ebe-ded6d862fe10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:19d2c6cd-ba91-4798-9ebe-ded6d862fe10', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:59,254 [237] [DEBUG] [app] Ending request: urn:request:e3f712ed-e8c4-4c70-8c35-a46b92092a47 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3f712ed-e8c4-4c70-8c35-a46b92092a47', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:03:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:03:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:03:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:03:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:03:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:03:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:03:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:04.936688+00:00 (in 4.998338 seconds)
exportactionlogsworker stdout | 2025-02-07 04:03:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:59 UTC)" (scheduled at 2025-02-07 04:03:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:03:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:03:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:04:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:04:00,096 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:04:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:04:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:23.420886+00:00 (in 20.001313 seconds)
repositorygcworker stdout | 2025-02-07 04:04:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:03 UTC)" (scheduled at 2025-02-07 04:04:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:04:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:04:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 3, 419787), True, datetime.datetime(2025, 2, 7, 4, 4, 3, 419787), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:04:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:04:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:04:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:04:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:04:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:33.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 04:04:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:33 UTC)" (scheduled at 2025-02-07 04:04:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:04:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 4, 3, 919990), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:04:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:04:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:04:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:04:04,375 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:04:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:04:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:52.937648+00:00 (in 48.000519 seconds)
exportactionlogsworker stdout | 2025-02-07 04:04:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:04 UTC)" (scheduled at 2025-02-07 04:04:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:04:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:04:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 4, 937430), True, datetime.datetime(2025, 2, 7, 4, 4, 4, 937430), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:04:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:04:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:04:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:04:05,376 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:04:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:04:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:18.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 04:04:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:18 UTC)" (scheduled at 2025-02-07 04:04:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:04:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:04:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 8, 732867), True, datetime.datetime(2025, 2, 7, 4, 4, 8, 732867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:04:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:04:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:04:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:04:09,455 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:04:09,978 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:04:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:04:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:24.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 04:04:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:40 UTC)" (scheduled at 2025-02-07 04:04:10.424250+00:00)
gcworker stdout | 2025-02-07 04:04:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:04:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900750430, None, 1, 0])
gcworker stdout | 2025-02-07 04:04:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:04:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:04:10,474 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:04:11,774 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:04:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:04:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:41.831521+00:00 (in 29.999525 seconds)
autopruneworker stdout | 2025-02-07 04:04:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:41 UTC)" (scheduled at 2025-02-07 04:04:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:04:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897451836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:04:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:04:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:04:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:04:12,646 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:04:12,729 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:04:13,818 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:14,142 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:04:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:04:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:29.225378+00:00 (in 14.999507 seconds)
quotatotalworker stdout | 2025-02-07 04:04:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:29 UTC)" (scheduled at 2025-02-07 04:04:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:04:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:04:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:04:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:04:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:04:14,236 [235] [DEBUG] [app] Starting request: urn:request:6ad42abd-ffba-4836-88f5-3b9fdb45525c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:14,236 [238] [DEBUG] [app] Starting request: urn:request:1067ec2c-52f9-4254-bf79-f43d53233583 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:04:14,239 [239] [DEBUG] [app] Starting request: urn:request:074d2431-ec59-4fb9-aef5-f4786b9bbc04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:14,239 [245] [DEBUG] [app] Starting request: urn:request:1e95f443-d9cf-4155-8bdc-5aaf7db88642 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:14,239 [239] [DEBUG] [app] Ending request: urn:request:074d2431-ec59-4fb9-aef5-f4786b9bbc04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:074d2431-ec59-4fb9-aef5-f4786b9bbc04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:04:14,239 [245] [DEBUG] [app] Ending request: urn:request:1e95f443-d9cf-4155-8bdc-5aaf7db88642 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e95f443-d9cf-4155-8bdc-5aaf7db88642', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:04:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:04:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:04:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:04:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:14,242 [237] [DEBUG] [app] Starting request: urn:request:4d5a972a-30bf-4f55-9c66-16e1ae0ff292 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:14,243 [237] [DEBUG] [app] Ending request: urn:request:4d5a972a-30bf-4f55-9c66-16e1ae0ff292 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d5a972a-30bf-4f55-9c66-16e1ae0ff292', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:04:14,243 [236] [DEBUG] [app] Starting request: urn:request:c772f185-0b7b-44a7-9e4a-f334f244e9ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:04:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:14,243 [236] [DEBUG] [app] Ending request: urn:request:c772f185-0b7b-44a7-9e4a-f334f244e9ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c772f185-0b7b-44a7-9e4a-f334f244e9ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:04:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:04:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:04:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:04:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:14,255 [235] [DEBUG] [app] Ending request: urn:request:6ad42abd-ffba-4836-88f5-3b9fdb45525c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6ad42abd-ffba-4836-88f5-3b9fdb45525c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:04:14,255 [238] [DEBUG] [app] Ending request: urn:request:1067ec2c-52f9-4254-bf79-f43d53233583 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1067ec2c-52f9-4254-bf79-f43d53233583', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:04:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:04:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
globalpromstats stdout | 2025-02-07 04:04:14,835 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:04:14,877 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:04:14,992 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:04:15,363 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:04:15,738 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:04:15,901 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:04:15,934 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:04:16,341 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:04:16,344 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:04:16,347 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:04:16,349 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:04:16,353 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:04:17,371 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:04:17,374 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:04:17,376 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:04:17,573 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:04:17,784 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:04:17,973 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:04:18,449 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:04:18,452 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:04:18,454 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:04:18,457 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:04:18,459 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:04:18,463 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:04:18,467 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:04:18,470 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:04:18,472 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:04:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:04:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:28.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 04:04:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:28 UTC)" (scheduled at 2025-02-07 04:04:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:04:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:04:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 18, 732882), True, datetime.datetime(2025, 2, 7, 4, 4, 18, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:04:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:04:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:04:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:04:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:04:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:03.419110+00:00 (in 39.997727 seconds)
repositorygcworker stdout | 2025-02-07 04:04:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:23 UTC)" (scheduled at 2025-02-07 04:04:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:04:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:04:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:04:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:04:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:40.424250+00:00 (in 16.000751 seconds)
gcworker stdout | 2025-02-07 04:04:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:54 UTC)" (scheduled at 2025-02-07 04:04:24.423055+00:00)
gcworker stdout | 2025-02-07 04:04:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:04:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:04:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:04:27,010 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:04:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:04:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:38.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 04:04:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:38 UTC)" (scheduled at 2025-02-07 04:04:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:04:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:04:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 28, 732868), True, datetime.datetime(2025, 2, 7, 4, 4, 28, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:04:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:04:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:04:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:04:28,940 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:04:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:04:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:44.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 04:04:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:44 UTC)" (scheduled at 2025-02-07 04:04:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:04:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:04:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:04:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:04:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:04:29,235 [238] [DEBUG] [app] Starting request: urn:request:c2299737-ccc9-4920-a955-6ee969c431ee (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:29,235 [235] [DEBUG] [app] Starting request: urn:request:96c2bad5-8e34-48bb-955f-bc8eec2853db (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:04:29,239 [244] [DEBUG] [app] Starting request: urn:request:fd5503d7-cd4a-4ec2-8399-215c373c3e6c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:29,239 [239] [DEBUG] [app] Starting request: urn:request:4d34a68d-c2e7-433e-a0e6-28969a8c39e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:29,239 [239] [DEBUG] [app] Ending request: urn:request:4d34a68d-c2e7-433e-a0e6-28969a8c39e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d34a68d-c2e7-433e-a0e6-28969a8c39e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:04:29,239 [244] [DEBUG] [app] Ending request: urn:request:fd5503d7-cd4a-4ec2-8399-215c373c3e6c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd5503d7-cd4a-4ec2-8399-215c373c3e6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:04:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:04:29,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:04:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:29,243 [236] [DEBUG] [app] Starting request: urn:request:a69d9291-411e-42f1-9327-1d969276be00 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:29,243 [237] [DEBUG] [app] Starting request: urn:request:6b2c93f8-f4d1-4e4f-81b5-f8879ee395e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:29,243 [237] [DEBUG] [app] Ending request: urn:request:6b2c93f8-f4d1-4e4f-81b5-f8879ee395e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b2c93f8-f4d1-4e4f-81b5-f8879ee395e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:04:29,243 [236] [DEBUG] [app] Ending request: urn:request:a69d9291-411e-42f1-9327-1d969276be00 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a69d9291-411e-42f1-9327-1d969276be00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:04:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:04:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:04:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:04:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:04:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:29,255 [238] [DEBUG] [app] Ending request: urn:request:c2299737-ccc9-4920-a955-6ee969c431ee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2299737-ccc9-4920-a955-6ee969c431ee', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:04:29,255 [235] [DEBUG] [app] Ending request: urn:request:96c2bad5-8e34-48bb-955f-bc8eec2853db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:96c2bad5-8e34-48bb-955f-bc8eec2853db', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:04:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:04:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
queuecleanupworker stdout | 2025-02-07 04:04:30,121 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:04:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:04:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:32.625945+00:00 (in 1.004274 seconds)
namespacegcworker stdout | 2025-02-07 04:04:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:31 UTC)" (scheduled at 2025-02-07 04:04:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:04:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:04:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 31, 621980), True, datetime.datetime(2025, 2, 7, 4, 4, 31, 621980), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:04:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:04:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:04:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:04:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:04:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:31.621131+00:00 (in 58.994651 seconds)
namespacegcworker stdout | 2025-02-07 04:04:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:32 UTC)" (scheduled at 2025-02-07 04:04:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:04:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:04:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:04:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:04:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:03.919286+00:00 (in 29.999523 seconds)
buildlogsarchiver stdout | 2025-02-07 04:04:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:03 UTC)" (scheduled at 2025-02-07 04:04:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:04:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 4, 33, 920037), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:04:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:04:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:04:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:04:34,411 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:04:35,384 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:56.713547+00:00 (in 18.997161 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:37 UTC)" (scheduled at 2025-02-07 04:04:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:04:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:04:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:48.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 04:04:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:48 UTC)" (scheduled at 2025-02-07 04:04:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:04:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:04:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 38, 732889), True, datetime.datetime(2025, 2, 7, 4, 4, 38, 732889), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:04:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:04:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:04:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:04:39,472 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:04:40,015 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:04:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:04:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:54.423055+00:00 (in 13.998354 seconds)
gcworker stdout | 2025-02-07 04:04:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:10 UTC)" (scheduled at 2025-02-07 04:04:40.424250+00:00)
gcworker stdout | 2025-02-07 04:04:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:04:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900780431, None, 1, 0])
gcworker stdout | 2025-02-07 04:04:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:04:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:04:40,511 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:04:41,810 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:04:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:04:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:11.831521+00:00 (in 29.999568 seconds)
autopruneworker stdout | 2025-02-07 04:04:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:11 UTC)" (scheduled at 2025-02-07 04:04:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:04:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897481836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:04:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:04:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:04:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:11 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:04:42,682 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:04:42,748 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:04:43,854 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:44,153 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:04:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:04:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:59.225378+00:00 (in 14.999514 seconds)
quotatotalworker stdout | 2025-02-07 04:04:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:59 UTC)" (scheduled at 2025-02-07 04:04:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:04:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:04:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:04:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:04:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:04:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:04:44,236 [238] [DEBUG] [app] Starting request: urn:request:08f984e4-1702-4bdd-aa3e-b788021221a8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:44,236 [235] [DEBUG] [app] Starting request: urn:request:a605738f-cd2e-4eb5-a4e2-cff1a590900c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:04:44,238 [239] [DEBUG] [app] Starting request: urn:request:1e035642-90f6-471d-afa3-d79ca606400d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:44,239 [246] [DEBUG] [app] Starting request: urn:request:845da2fb-af31-44ac-8281-a75f8fa23a51 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:44,239 [239] [DEBUG] [app] Ending request: urn:request:1e035642-90f6-471d-afa3-d79ca606400d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e035642-90f6-471d-afa3-d79ca606400d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:04:44,239 [246] [DEBUG] [app] Ending request: urn:request:845da2fb-af31-44ac-8281-a75f8fa23a51 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:845da2fb-af31-44ac-8281-a75f8fa23a51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:04:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:04:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:04:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:44,242 [238] [DEBUG] [app] Starting request: urn:request:13dd46ed-8d40-4882-aea1-7f1cc1151726 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:44,242 [238] [DEBUG] [app] Ending request: urn:request:13dd46ed-8d40-4882-aea1-7f1cc1151726 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:13dd46ed-8d40-4882-aea1-7f1cc1151726', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:04:44,242 [236] [DEBUG] [app] Starting request: urn:request:58ae9a7f-bd35-499a-9db8-289c0e91201d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:04:44,242 [236] [DEBUG] [app] Ending request: urn:request:58ae9a7f-bd35-499a-9db8-289c0e91201d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58ae9a7f-bd35-499a-9db8-289c0e91201d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:04:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:04:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:04:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:04:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:44,253 [238] [DEBUG] [app] Ending request: urn:request:08f984e4-1702-4bdd-aa3e-b788021221a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08f984e4-1702-4bdd-aa3e-b788021221a8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:04:44,254 [235] [DEBUG] [app] Ending request: urn:request:a605738f-cd2e-4eb5-a4e2-cff1a590900c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a605738f-cd2e-4eb5-a4e2-cff1a590900c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:04:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:04:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
globalpromstats stdout | 2025-02-07 04:04:44,868 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:04:44,913 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:04:45,026 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:04:45,385 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:04:45,773 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:04:45,930 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:04:45,971 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:04:46,350 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:04:46,353 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:04:46,355 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:04:46,358 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:04:46,360 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:04:47,378 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:04:47,382 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:04:47,385 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:04:47,607 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:04:47,812 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:04:48,009 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:04:48,458 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:04:48,463 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:04:48,465 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:04:48,468 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:04:48,471 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:04:48,473 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:04:48,477 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:04:48,480 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:04:48,483 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:04:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:04:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:50.814677+00:00 (in 2.082077 seconds)
notificationworker stdout | 2025-02-07 04:04:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:58 UTC)" (scheduled at 2025-02-07 04:04:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:04:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:04:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 48, 732896), True, datetime.datetime(2025, 2, 7, 4, 4, 48, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:04:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:04:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:04:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:04:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:04:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:04:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:58.732147+00:00 (in 7.916994 seconds)
notificationworker stdout | 2025-02-07 04:04:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:50 UTC)" (scheduled at 2025-02-07 04:04:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:04:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:04:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:04:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:04:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:04:59.937899+00:00 (in 6.999765 seconds)
exportactionlogsworker stdout | 2025-02-07 04:04:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:09:52 UTC)" (scheduled at 2025-02-07 04:04:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:04:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 4, 52, 938443), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:04:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 4, 52, 938443), True, datetime.datetime(2025, 2, 7, 4, 4, 52, 938443), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:04:52,946 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 4, 52, 938443), True, datetime.datetime(2025, 2, 7, 4, 4, 52, 938443), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 4, 52, 938443), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:04:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:04:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:09:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:04:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:04:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:10.424250+00:00 (in 16.000729 seconds)
gcworker stdout | 2025-02-07 04:04:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:24 UTC)" (scheduled at 2025-02-07 04:04:54.423055+00:00)
gcworker stdout | 2025-02-07 04:04:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:04:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:04:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:04:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:56.118855+00:00 (in 59.999477 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:04:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:56 UTC)" (scheduled at 2025-02-07 04:04:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:04:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:04:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:04:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:37.715878+00:00 (in 41.001874 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:56 UTC)" (scheduled at 2025-02-07 04:04:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:04:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:04:57,046 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:04:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:04:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:08.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 04:04:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:08 UTC)" (scheduled at 2025-02-07 04:04:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:04:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:04:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 4, 58, 732823), True, datetime.datetime(2025, 2, 7, 4, 4, 58, 732823), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:04:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:04:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:04:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:04:58,969 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:04:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:04:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:14.225378+00:00 (in 14.999511 seconds)
quotatotalworker stdout | 2025-02-07 04:04:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:14 UTC)" (scheduled at 2025-02-07 04:04:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:04:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:04:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:04:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:04:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:04:59,235 [236] [DEBUG] [app] Starting request: urn:request:917af397-ea62-46fc-9bb4-7aaea955d5d8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:59,235 [237] [DEBUG] [app] Starting request: urn:request:23ff8868-f288-4128-8a65-d7759a4a7dcd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:04:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:04:59,238 [239] [DEBUG] [app] Starting request: urn:request:28dc8b6e-2c51-49e4-a714-d006c6a71d00 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:59,238 [245] [DEBUG] [app] Starting request: urn:request:bee24f39-b6f9-45b9-96aa-14b613939b8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:04:59,238 [245] [DEBUG] [app] Ending request: urn:request:bee24f39-b6f9-45b9-96aa-14b613939b8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bee24f39-b6f9-45b9-96aa-14b613939b8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:04:59,238 [239] [DEBUG] [app] Ending request: urn:request:28dc8b6e-2c51-49e4-a714-d006c6a71d00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28dc8b6e-2c51-49e4-a714-d006c6a71d00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:04:59,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:04:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:04:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:04:59,241 [238] [DEBUG] [app] Starting request: urn:request:6e89da40-e905-4a8d-ba84-5f46ea677abf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:59,241 [238] [DEBUG] [app] Ending request: urn:request:6e89da40-e905-4a8d-ba84-5f46ea677abf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e89da40-e905-4a8d-ba84-5f46ea677abf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:04:59,241 [236] [DEBUG] [app] Starting request: urn:request:a6db1dfd-9601-473e-9901-fc44aee1fa21 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:04:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:04:59,242 [236] [DEBUG] [app] Ending request: urn:request:a6db1dfd-9601-473e-9901-fc44aee1fa21 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6db1dfd-9601-473e-9901-fc44aee1fa21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:04:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:04:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:04:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:04:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:04:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:04:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:04:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:04:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:04:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:04:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:04:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:04:59,253 [236] [DEBUG] [app] Ending request: urn:request:917af397-ea62-46fc-9bb4-7aaea955d5d8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:917af397-ea62-46fc-9bb4-7aaea955d5d8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:59,253 [237] [DEBUG] [app] Ending request: urn:request:23ff8868-f288-4128-8a65-d7759a4a7dcd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23ff8868-f288-4128-8a65-d7759a4a7dcd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:04:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:04:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:04:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:04:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:04:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:04:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:04:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:04:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:04.936688+00:00 (in 4.998271 seconds)
exportactionlogsworker stdout | 2025-02-07 04:04:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:59 UTC)" (scheduled at 2025-02-07 04:04:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:04:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:04:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:05:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:05:00,157 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:05:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:05:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:23.420886+00:00 (in 20.001275 seconds)
repositorygcworker stdout | 2025-02-07 04:05:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:03 UTC)" (scheduled at 2025-02-07 04:05:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:05:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:05:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 3, 419895), True, datetime.datetime(2025, 2, 7, 4, 5, 3, 419895), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:05:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:05:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:05:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:05:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:05:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:33.919286+00:00 (in 29.999487 seconds)
buildlogsarchiver stdout | 2025-02-07 04:05:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:33 UTC)" (scheduled at 2025-02-07 04:05:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:05:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 5, 3, 920112), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:05:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:05:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:05:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:05:04,419 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:05:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:05:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:59.937899+00:00 (in 55.000745 seconds)
exportactionlogsworker stdout | 2025-02-07 04:05:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:04 UTC)" (scheduled at 2025-02-07 04:05:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:05:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:05:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 4, 937467), True, datetime.datetime(2025, 2, 7, 4, 5, 4, 937467), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:05:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:05:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:05:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:05:05,420 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:05:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:05:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:18.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:05:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:18 UTC)" (scheduled at 2025-02-07 04:05:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:05:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:05:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 8, 732797), True, datetime.datetime(2025, 2, 7, 4, 5, 8, 732797), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:05:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:05:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:05:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:05:09,505 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:05:10,026 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:05:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:05:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:24.423055+00:00 (in 13.998367 seconds)
gcworker stdout | 2025-02-07 04:05:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:40 UTC)" (scheduled at 2025-02-07 04:05:10.424250+00:00)
gcworker stdout | 2025-02-07 04:05:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:05:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900810430, None, 1, 0])
gcworker stdout | 2025-02-07 04:05:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:05:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:05:10,531 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:05:11,819 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
autopruneworker stdout | 2025-02-07 04:05:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:05:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:41.831521+00:00 (in 29.999484 seconds)
autopruneworker stdout | 2025-02-07 04:05:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:41 UTC)" (scheduled at 2025-02-07 04:05:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:05:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897511836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:05:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:05:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:05:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:41 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:05:12,717 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:05:12,774 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:05:13,890 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:14,184 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:05:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:05:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:29.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:05:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:29 UTC)" (scheduled at 2025-02-07 04:05:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:05:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:05:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:05:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:05:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:05:14,235 [238] [DEBUG] [app] Starting request: urn:request:f655a1fb-7dcc-4be9-af04-b2ef43945ca9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:14,235 [235] [DEBUG] [app] Starting request: urn:request:0f7dac45-6c7a-44db-a6a2-2939b294342d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:05:14,238 [244] [DEBUG] [app] Starting request: urn:request:50087ed5-78c6-4434-9d94-b173d4ea5e54 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:05:14,238 [245] [DEBUG] [app] Starting request: urn:request:30a25519-e025-416c-a371-e7d75f581aca (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:05:14,239 [245] [DEBUG] [app] Ending request: urn:request:30a25519-e025-416c-a371-e7d75f581aca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30a25519-e025-416c-a371-e7d75f581aca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:05:14,239 [244] [DEBUG] [app] Ending request: urn:request:50087ed5-78c6-4434-9d94-b173d4ea5e54 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50087ed5-78c6-4434-9d94-b173d4ea5e54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:05:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:05:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:14,241 [235] [DEBUG] [app] Starting request: urn:request:9972d0d2-3ba2-4844-a415-14fc52059918 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:05:14,242 [237] [DEBUG] [app] Starting request: urn:request:2c474ac0-6ae0-4e4d-b26b-5490ed606bd6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:05:14,242 [235] [DEBUG] [app] Ending request: urn:request:9972d0d2-3ba2-4844-a415-14fc52059918 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9972d0d2-3ba2-4844-a415-14fc52059918', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:14,242 [237] [DEBUG] [app] Ending request: urn:request:2c474ac0-6ae0-4e4d-b26b-5490ed606bd6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2c474ac0-6ae0-4e4d-b26b-5490ed606bd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:05:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:05:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:14,254 [235] [DEBUG] [app] Ending request: urn:request:0f7dac45-6c7a-44db-a6a2-2939b294342d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f7dac45-6c7a-44db-a6a2-2939b294342d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:14,254 [238] [DEBUG] [app] Ending request: urn:request:f655a1fb-7dcc-4be9-af04-b2ef43945ca9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f655a1fb-7dcc-4be9-af04-b2ef43945ca9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:05:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
globalpromstats stdout | 2025-02-07 04:05:14,901 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:05:14,944 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:05:15,041 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:05:15,410 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:05:15,810 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:05:15,966 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:05:15,980 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:05:16,360 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:05:16,363 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:05:16,366 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:05:16,369 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:05:16,372 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:05:17,386 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:05:17,389 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:05:17,393 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:05:17,643 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:05:17,849 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:05:18,046 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:05:18,468 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:05:18,471 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:05:18,477 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:05:18,481 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:05:18,484 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:05:18,487 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:05:18,490 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:05:18,492 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:05:18,496 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:05:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:05:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:28.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 04:05:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:28 UTC)" (scheduled at 2025-02-07 04:05:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:05:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:05:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 18, 732898), True, datetime.datetime(2025, 2, 7, 4, 5, 18, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:05:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:05:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:05:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:05:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:05:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:03.419110+00:00 (in 39.997752 seconds)
repositorygcworker stdout | 2025-02-07 04:05:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:23 UTC)" (scheduled at 2025-02-07 04:05:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:05:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:05:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:05:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:05:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:40.424250+00:00 (in 16.000749 seconds)
gcworker stdout | 2025-02-07 04:05:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:54 UTC)" (scheduled at 2025-02-07 04:05:24.423055+00:00)
gcworker stdout | 2025-02-07 04:05:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:05:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:05:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:05:27,082 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:05:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:05:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:38.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 04:05:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:38 UTC)" (scheduled at 2025-02-07 04:05:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:05:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:05:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 28, 732919), True, datetime.datetime(2025, 2, 7, 4, 5, 28, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:05:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:05:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:05:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:05:28,978 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:05:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:05:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:44.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 04:05:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:44 UTC)" (scheduled at 2025-02-07 04:05:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:05:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:05:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:05:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:05:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:05:29,235 [237] [DEBUG] [app] Starting request: urn:request:adfd5841-80bf-4e3b-a541-33282c755426 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:29,235 [238] [DEBUG] [app] Starting request: urn:request:e8367cff-0cd1-44da-8c49-8178c4237a6c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:05:29,238 [246] [DEBUG] [app] Starting request: urn:request:15431166-73c1-4f64-9d93-89eaf6d148b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:05:29,238 [239] [DEBUG] [app] Starting request: urn:request:bf736969-c6dc-42e7-b6ff-9f72b9e22903 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:05:29,238 [246] [DEBUG] [app] Ending request: urn:request:15431166-73c1-4f64-9d93-89eaf6d148b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:15431166-73c1-4f64-9d93-89eaf6d148b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:05:29,238 [239] [DEBUG] [app] Ending request: urn:request:bf736969-c6dc-42e7-b6ff-9f72b9e22903 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf736969-c6dc-42e7-b6ff-9f72b9e22903', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:05:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:05:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:29,241 [237] [DEBUG] [app] Starting request: urn:request:d4a6ca77-bdb6-4136-af40-23b258f3904c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:05:29,241 [237] [DEBUG] [app] Ending request: urn:request:d4a6ca77-bdb6-4136-af40-23b258f3904c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4a6ca77-bdb6-4136-af40-23b258f3904c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:29,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:29,241 [235] [DEBUG] [app] Starting request: urn:request:5dc90a58-70a2-4bb6-b9aa-ba3f8fa45dd3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:05:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:29,242 [235] [DEBUG] [app] Ending request: urn:request:5dc90a58-70a2-4bb6-b9aa-ba3f8fa45dd3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5dc90a58-70a2-4bb6-b9aa-ba3f8fa45dd3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:05:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:29,253 [237] [DEBUG] [app] Ending request: urn:request:adfd5841-80bf-4e3b-a541-33282c755426 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:adfd5841-80bf-4e3b-a541-33282c755426', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:29,253 [238] [DEBUG] [app] Ending request: urn:request:e8367cff-0cd1-44da-8c49-8178c4237a6c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e8367cff-0cd1-44da-8c49-8178c4237a6c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:05:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
queuecleanupworker stdout | 2025-02-07 04:05:30,193 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:05:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:05:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:32.625945+00:00 (in 1.004320 seconds)
namespacegcworker stdout | 2025-02-07 04:05:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:31 UTC)" (scheduled at 2025-02-07 04:05:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:05:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:05:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 31, 621902), True, datetime.datetime(2025, 2, 7, 4, 5, 31, 621902), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:05:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:05:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:05:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:05:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:05:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:31.621131+00:00 (in 58.994713 seconds)
namespacegcworker stdout | 2025-02-07 04:05:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:32 UTC)" (scheduled at 2025-02-07 04:05:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:05:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:05:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:05:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:05:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:03.919286+00:00 (in 29.999498 seconds)
buildlogsarchiver stdout | 2025-02-07 04:05:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:03 UTC)" (scheduled at 2025-02-07 04:05:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:05:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 5, 33, 920062), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:05:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:05:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:05:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:05:34,449 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:05:35,435 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:56.713547+00:00 (in 18.997210 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:37 UTC)" (scheduled at 2025-02-07 04:05:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:05:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:05:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:48.732147+00:00 (in 9.999501 seconds)
notificationworker stdout | 2025-02-07 04:05:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:48 UTC)" (scheduled at 2025-02-07 04:05:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:05:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:05:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 38, 732948), True, datetime.datetime(2025, 2, 7, 4, 5, 38, 732948), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:05:38,743 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:05:38,743 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:05:38,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:05:39,541 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:05:40,062 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:05:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:05:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:54.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 04:05:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:10 UTC)" (scheduled at 2025-02-07 04:05:40.424250+00:00)
gcworker stdout | 2025-02-07 04:05:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:05:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900840431, None, 1, 0])
gcworker stdout | 2025-02-07 04:05:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:05:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:05:40,567 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:05:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:05:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:11.831521+00:00 (in 29.999481 seconds)
autopruneworker stdout | 2025-02-07 04:05:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:11 UTC)" (scheduled at 2025-02-07 04:05:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:05:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897541836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:05:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:05:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:05:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:05:41,853 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:05:42,748 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:05:42,811 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:05:43,912 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:44,221 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
quotatotalworker stdout | 2025-02-07 04:05:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:05:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:59.225378+00:00 (in 14.999576 seconds)
quotatotalworker stdout | 2025-02-07 04:05:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:59 UTC)" (scheduled at 2025-02-07 04:05:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:05:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:05:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:05:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:05:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:05:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:05:44,236 [237] [DEBUG] [app] Starting request: urn:request:ed88e711-962f-447d-822c-01180a1b08da (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:44,236 [235] [DEBUG] [app] Starting request: urn:request:d31a02c4-6042-49ff-951a-4ec697606c49 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:05:44,239 [246] [DEBUG] [app] Starting request: urn:request:f8f3271d-7420-4c2f-851b-6828e8236242 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:05:44,239 [246] [DEBUG] [app] Ending request: urn:request:f8f3271d-7420-4c2f-851b-6828e8236242 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f8f3271d-7420-4c2f-851b-6828e8236242', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:05:44,239 [245] [DEBUG] [app] Starting request: urn:request:8b5f2290-ca8a-4f03-b08a-10253a708326 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:05:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:05:44,239 [245] [DEBUG] [app] Ending request: urn:request:8b5f2290-ca8a-4f03-b08a-10253a708326 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b5f2290-ca8a-4f03-b08a-10253a708326', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:05:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:44,242 [238] [DEBUG] [app] Starting request: urn:request:e924f65a-9fa8-4e2e-a8d4-1c63b09f41d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:05:44,243 [238] [DEBUG] [app] Ending request: urn:request:e924f65a-9fa8-4e2e-a8d4-1c63b09f41d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e924f65a-9fa8-4e2e-a8d4-1c63b09f41d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:05:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:44,243 [236] [DEBUG] [app] Starting request: urn:request:d5cec7a3-5db2-4ac0-9b60-9c87be9f1ce4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:05:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:44,244 [236] [DEBUG] [app] Ending request: urn:request:d5cec7a3-5db2-4ac0-9b60-9c87be9f1ce4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5cec7a3-5db2-4ac0-9b60-9c87be9f1ce4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:05:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:44,254 [237] [DEBUG] [app] Ending request: urn:request:ed88e711-962f-447d-822c-01180a1b08da (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed88e711-962f-447d-822c-01180a1b08da', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:05:44,256 [235] [DEBUG] [app] Ending request: urn:request:d31a02c4-6042-49ff-951a-4ec697606c49 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d31a02c4-6042-49ff-951a-4ec697606c49', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
globalpromstats stdout | 2025-02-07 04:05:44,916 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:05:44,979 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:05:45,061 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:05:45,444 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:05:45,836 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:05:46,002 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:05:46,014 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:05:46,368 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:05:46,371 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:05:46,376 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:05:46,378 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:05:46,381 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:05:47,393 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:05:47,397 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:05:47,401 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:05:47,679 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:05:47,885 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:05:48,078 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:05:48,475 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:05:48,483 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:05:48,486 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:05:48,490 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:05:48,495 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:05:48,498 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:05:48,501 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:05:48,504 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:05:48,506 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:05:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:05:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:50.814677+00:00 (in 2.082057 seconds)
notificationworker stdout | 2025-02-07 04:05:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:58 UTC)" (scheduled at 2025-02-07 04:05:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:05:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:05:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 48, 732826), True, datetime.datetime(2025, 2, 7, 4, 5, 48, 732826), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:05:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:05:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:05:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:05:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:05:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:05:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:05:58.732147+00:00 (in 7.916975 seconds)
notificationworker stdout | 2025-02-07 04:05:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:50 UTC)" (scheduled at 2025-02-07 04:05:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:05:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:05:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:05:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:05:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:10.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 04:05:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:24 UTC)" (scheduled at 2025-02-07 04:05:54.423055+00:00)
gcworker stdout | 2025-02-07 04:05:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:05:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:05:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:05:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:56.118855+00:00 (in 59.999473 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:05:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:56 UTC)" (scheduled at 2025-02-07 04:05:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:05:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:05:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:05:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:37.715878+00:00 (in 41.001846 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:56 UTC)" (scheduled at 2025-02-07 04:05:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:05:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:05:57,118 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:05:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:05:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:08.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 04:05:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:08 UTC)" (scheduled at 2025-02-07 04:05:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:05:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:05:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 5, 58, 732888), True, datetime.datetime(2025, 2, 7, 4, 5, 58, 732888), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:05:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:05:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:05:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:05:59,014 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:05:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:05:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:14.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 04:05:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:14 UTC)" (scheduled at 2025-02-07 04:05:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:05:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:05:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:05:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:05:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:05:59,236 [238] [DEBUG] [app] Starting request: urn:request:a2ec13c1-6d88-4762-8b31-683d2d9f7417 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:59,236 [235] [DEBUG] [app] Starting request: urn:request:b761436c-4d35-46eb-8677-82cb16ca6df7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:05:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:05:59,239 [246] [DEBUG] [app] Starting request: urn:request:e74a2ee5-ba73-4dec-bc84-963a9db6017a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:05:59,239 [245] [DEBUG] [app] Starting request: urn:request:d8e5b63e-3a79-4c97-bf18-50dd8abb9597 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:05:59,239 [246] [DEBUG] [app] Ending request: urn:request:e74a2ee5-ba73-4dec-bc84-963a9db6017a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e74a2ee5-ba73-4dec-bc84-963a9db6017a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:05:59,239 [245] [DEBUG] [app] Ending request: urn:request:d8e5b63e-3a79-4c97-bf18-50dd8abb9597 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8e5b63e-3a79-4c97-bf18-50dd8abb9597', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:05:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:05:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:05:59,242 [237] [DEBUG] [app] Starting request: urn:request:f9352c12-16cf-4789-9eef-579fc3e20c05 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:05:59,243 [235] [DEBUG] [app] Starting request: urn:request:3792e357-3171-499d-a3d7-7afd59d160fe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:05:59,243 [237] [DEBUG] [app] Ending request: urn:request:f9352c12-16cf-4789-9eef-579fc3e20c05 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9352c12-16cf-4789-9eef-579fc3e20c05', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:59,243 [235] [DEBUG] [app] Ending request: urn:request:3792e357-3171-499d-a3d7-7afd59d160fe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3792e357-3171-499d-a3d7-7afd59d160fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:05:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:05:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:05:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:05:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:05:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:05:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:05:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:05:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:05:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:05:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:05:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:05:59,255 [238] [DEBUG] [app] Ending request: urn:request:a2ec13c1-6d88-4762-8b31-683d2d9f7417 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2ec13c1-6d88-4762-8b31-683d2d9f7417', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:59,255 [235] [DEBUG] [app] Ending request: urn:request:b761436c-4d35-46eb-8677-82cb16ca6df7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b761436c-4d35-46eb-8677-82cb16ca6df7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:05:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:05:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:05:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:05:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:05:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:05:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:05:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:04.936688+00:00 (in 4.998285 seconds)
exportactionlogsworker stdout | 2025-02-07 04:05:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:59 UTC)" (scheduled at 2025-02-07 04:05:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:05:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:05:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:06:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:06:00,206 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:06:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:06:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:23.420886+00:00 (in 20.001279 seconds)
repositorygcworker stdout | 2025-02-07 04:06:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:03 UTC)" (scheduled at 2025-02-07 04:06:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:06:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:06:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 3, 419826), True, datetime.datetime(2025, 2, 7, 4, 6, 3, 419826), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:06:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:06:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:06:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:06:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:06:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:33.919286+00:00 (in 29.999522 seconds)
buildlogsarchiver stdout | 2025-02-07 04:06:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:33 UTC)" (scheduled at 2025-02-07 04:06:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:06:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 6, 3, 920043), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:06:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:06:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:06:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:06:04,485 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:06:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:06:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:59.937899+00:00 (in 55.000762 seconds)
exportactionlogsworker stdout | 2025-02-07 04:06:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:04 UTC)" (scheduled at 2025-02-07 04:06:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:06:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:06:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 4, 937483), True, datetime.datetime(2025, 2, 7, 4, 6, 4, 937483), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:06:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:06:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:06:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:06:05,462 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:06:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:06:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:18.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 04:06:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:18 UTC)" (scheduled at 2025-02-07 04:06:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:06:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:06:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 8, 732878), True, datetime.datetime(2025, 2, 7, 4, 6, 8, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:06:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:06:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:06:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:06:09,561 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:06:10,099 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:06:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:06:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:24.423055+00:00 (in 13.998314 seconds)
gcworker stdout | 2025-02-07 04:06:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:40 UTC)" (scheduled at 2025-02-07 04:06:10.424250+00:00)
gcworker stdout | 2025-02-07 04:06:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:06:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900870431, None, 1, 0])
gcworker stdout | 2025-02-07 04:06:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:06:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:06:10,591 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:06:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:06:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:41.831521+00:00 (in 29.999516 seconds)
autopruneworker stdout | 2025-02-07 04:06:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:41 UTC)" (scheduled at 2025-02-07 04:06:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:06:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897571836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:06:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:06:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:06:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:06:11,890 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:06:12,778 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:06:12,847 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:06:13,948 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:06:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:06:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:29.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 04:06:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:29 UTC)" (scheduled at 2025-02-07 04:06:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:06:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:06:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:06:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:06:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:06:14,236 [238] [DEBUG] [app] Starting request: urn:request:7ce5d18c-2940-487e-b94a-5a843a9978e5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:14,236 [235] [DEBUG] [app] Starting request: urn:request:06eea4c0-3b1a-439a-bff8-888fcacbf86b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:06:14,238 [239] [DEBUG] [app] Starting request: urn:request:5562ad1c-fa84-49b9-9c60-92bd51860068 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:14,238 [244] [DEBUG] [app] Starting request: urn:request:56d662e2-aa00-4c75-a24f-ea314b03a507 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:14,239 [244] [DEBUG] [app] Ending request: urn:request:56d662e2-aa00-4c75-a24f-ea314b03a507 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56d662e2-aa00-4c75-a24f-ea314b03a507', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:06:14,239 [239] [DEBUG] [app] Ending request: urn:request:5562ad1c-fa84-49b9-9c60-92bd51860068 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5562ad1c-fa84-49b9-9c60-92bd51860068', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:06:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:06:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:14,242 [235] [DEBUG] [app] Starting request: urn:request:e9870696-00c2-4ab9-a594-d0b6fcf88088 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:06:14,242 [236] [DEBUG] [app] Starting request: urn:request:f6bf0e24-86e8-4925-a0c4-c48f47c790a7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:06:14,242 [235] [DEBUG] [app] Ending request: urn:request:e9870696-00c2-4ab9-a594-d0b6fcf88088 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e9870696-00c2-4ab9-a594-d0b6fcf88088', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:06:14,242 [236] [DEBUG] [app] Ending request: urn:request:f6bf0e24-86e8-4925-a0c4-c48f47c790a7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6bf0e24-86e8-4925-a0c4-c48f47c790a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:06:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:06:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:06:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:14,254 [235] [DEBUG] [app] Ending request: urn:request:06eea4c0-3b1a-439a-bff8-888fcacbf86b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:06eea4c0-3b1a-439a-bff8-888fcacbf86b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:14,254 [238] [DEBUG] [app] Ending request: urn:request:7ce5d18c-2940-487e-b94a-5a843a9978e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7ce5d18c-2940-487e-b94a-5a843a9978e5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:06:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:06:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:06:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:14,257 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:06:14,932 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:06:15,016 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:06:15,085 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:06:15,472 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:06:15,864 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:06:16,039 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:06:16,048 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:06:16,376 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:06:16,379 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:06:16,383 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:06:16,387 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:06:16,389 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:06:17,400 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:06:17,404 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:06:17,407 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:06:17,697 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:06:17,921 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:06:18,115 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:06:18,484 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:06:18,494 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:06:18,498 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:06:18,500 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:06:18,504 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:06:18,510 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:06:18,513 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:06:18,516 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:06:18,519 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:06:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:06:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:28.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:06:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:28 UTC)" (scheduled at 2025-02-07 04:06:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:06:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:06:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 18, 732873), True, datetime.datetime(2025, 2, 7, 4, 6, 18, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:06:18,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:06:18,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:06:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:06:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:06:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:03.419110+00:00 (in 39.997735 seconds)
repositorygcworker stdout | 2025-02-07 04:06:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:23 UTC)" (scheduled at 2025-02-07 04:06:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:06:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:06:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:06:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:06:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:40.424250+00:00 (in 16.000702 seconds)
gcworker stdout | 2025-02-07 04:06:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:54 UTC)" (scheduled at 2025-02-07 04:06:24.423055+00:00)
gcworker stdout | 2025-02-07 04:06:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:06:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:06:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:06:27,155 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:06:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:06:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:38.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 04:06:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:38 UTC)" (scheduled at 2025-02-07 04:06:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:06:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:06:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 28, 732897), True, datetime.datetime(2025, 2, 7, 4, 6, 28, 732897), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:06:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:06:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:06:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:06:29,051 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:06:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:06:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:44.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 04:06:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:44 UTC)" (scheduled at 2025-02-07 04:06:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:06:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:06:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:06:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:06:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:06:29,235 [238] [DEBUG] [app] Starting request: urn:request:7b8c62fc-e4d6-41f9-9691-19b238ab5c12 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:29,235 [235] [DEBUG] [app] Starting request: urn:request:308f7b0f-dc8f-4585-ba0a-d98083c0ceff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:06:29,238 [239] [DEBUG] [app] Starting request: urn:request:36ef7fc0-565b-43e3-9527-3d860bfc38f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:29,238 [246] [DEBUG] [app] Starting request: urn:request:c43a53f2-846d-48da-9075-5fa2e9a34c76 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:29,238 [239] [DEBUG] [app] Ending request: urn:request:36ef7fc0-565b-43e3-9527-3d860bfc38f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36ef7fc0-565b-43e3-9527-3d860bfc38f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:06:29,238 [246] [DEBUG] [app] Ending request: urn:request:c43a53f2-846d-48da-9075-5fa2e9a34c76 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c43a53f2-846d-48da-9075-5fa2e9a34c76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:06:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:06:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:06:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:29,241 [237] [DEBUG] [app] Starting request: urn:request:0d35ed47-be02-4c60-875a-213787076c83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:06:29,242 [237] [DEBUG] [app] Ending request: urn:request:0d35ed47-be02-4c60-875a-213787076c83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d35ed47-be02-4c60-875a-213787076c83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:06:29,242 [238] [DEBUG] [app] Starting request: urn:request:181dbdd3-39ef-43f2-9295-26dbb55c7a83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:06:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:29,242 [238] [DEBUG] [app] Ending request: urn:request:181dbdd3-39ef-43f2-9295-26dbb55c7a83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:181dbdd3-39ef-43f2-9295-26dbb55c7a83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:06:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:06:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:29,253 [238] [DEBUG] [app] Ending request: urn:request:7b8c62fc-e4d6-41f9-9691-19b238ab5c12 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7b8c62fc-e4d6-41f9-9691-19b238ab5c12', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:29,254 [235] [DEBUG] [app] Ending request: urn:request:308f7b0f-dc8f-4585-ba0a-d98083c0ceff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:308f7b0f-dc8f-4585-ba0a-d98083c0ceff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:06:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:06:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:06:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 04:06:30,240 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:06:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:06:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:32.625945+00:00 (in 1.004347 seconds)
namespacegcworker stdout | 2025-02-07 04:06:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:31 UTC)" (scheduled at 2025-02-07 04:06:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:06:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:06:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 31, 621873), True, datetime.datetime(2025, 2, 7, 4, 6, 31, 621873), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:06:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:06:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:06:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:06:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:06:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:13.624301+00:00 (in 40.997888 seconds)
namespacegcworker stdout | 2025-02-07 04:06:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:32 UTC)" (scheduled at 2025-02-07 04:06:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:06:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:06:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:06:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:06:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:03.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 04:06:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:03 UTC)" (scheduled at 2025-02-07 04:06:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:06:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 6, 33, 919997), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:06:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:06:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:06:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:06:34,495 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:06:35,498 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:56.713547+00:00 (in 18.997183 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:37 UTC)" (scheduled at 2025-02-07 04:06:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:06:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:06:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:48.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:06:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:48 UTC)" (scheduled at 2025-02-07 04:06:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:06:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:06:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 38, 732848), True, datetime.datetime(2025, 2, 7, 4, 6, 38, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:06:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:06:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:06:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:06:39,579 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:06:40,119 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:06:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:06:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:54.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 04:06:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:10 UTC)" (scheduled at 2025-02-07 04:06:40.424250+00:00)
gcworker stdout | 2025-02-07 04:06:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:06:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900900430, None, 1, 0])
gcworker stdout | 2025-02-07 04:06:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:06:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:06:40,628 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:06:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:06:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:11.831521+00:00 (in 29.999550 seconds)
autopruneworker stdout | 2025-02-07 04:06:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:11 UTC)" (scheduled at 2025-02-07 04:06:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:06:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897601836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:06:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:06:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:06:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:06:41,912 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:06:42,814 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:06:42,853 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:06:43,967 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:06:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:06:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:59.225378+00:00 (in 14.999532 seconds)
quotatotalworker stdout | 2025-02-07 04:06:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:59 UTC)" (scheduled at 2025-02-07 04:06:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:06:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:06:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:06:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:06:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:06:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:06:44,235 [235] [DEBUG] [app] Starting request: urn:request:4b996dc5-6a7a-441a-8ac4-b9187d52fd88 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:44,235 [238] [DEBUG] [app] Starting request: urn:request:0470ccbd-40f5-4274-932c-e9fcb08b45a9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:06:44,238 [245] [DEBUG] [app] Starting request: urn:request:60e26903-3299-418d-939b-75264df3c0a9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:44,239 [245] [DEBUG] [app] Ending request: urn:request:60e26903-3299-418d-939b-75264df3c0a9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60e26903-3299-418d-939b-75264df3c0a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:06:44,239 [246] [DEBUG] [app] Starting request: urn:request:b441a7d7-23e2-40e3-a284-12acd47c7dd2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:06:44,239 [246] [DEBUG] [app] Ending request: urn:request:b441a7d7-23e2-40e3-a284-12acd47c7dd2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b441a7d7-23e2-40e3-a284-12acd47c7dd2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:06:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:06:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:44,242 [238] [DEBUG] [app] Starting request: urn:request:7d3a078e-124f-420c-bcaf-37c3551dc1f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:06:44,242 [238] [DEBUG] [app] Ending request: urn:request:7d3a078e-124f-420c-bcaf-37c3551dc1f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d3a078e-124f-420c-bcaf-37c3551dc1f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:06:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:44,243 [237] [DEBUG] [app] Starting request: urn:request:51db2c2b-55ce-4c36-a074-6845cc0ef839 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:06:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:44,244 [237] [DEBUG] [app] Ending request: urn:request:51db2c2b-55ce-4c36-a074-6845cc0ef839 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:51db2c2b-55ce-4c36-a074-6845cc0ef839', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:06:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:06:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:44,255 [238] [DEBUG] [app] Ending request: urn:request:0470ccbd-40f5-4274-932c-e9fcb08b45a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0470ccbd-40f5-4274-932c-e9fcb08b45a9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:06:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:06:44,255 [235] [DEBUG] [app] Ending request: urn:request:4b996dc5-6a7a-441a-8ac4-b9187d52fd88 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4b996dc5-6a7a-441a-8ac4-b9187d52fd88', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 04:06:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:44,290 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:06:44,969 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:06:45,026 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:06:45,114 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:06:45,505 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:06:45,879 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:06:46,057 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:06:46,084 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:06:46,386 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:06:46,389 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:06:46,392 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:06:46,395 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:06:46,398 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:06:47,407 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:06:47,411 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:06:47,414 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:06:47,711 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:06:47,945 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:06:48,151 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:06:48,496 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:06:48,504 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:06:48,507 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:06:48,509 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:06:48,512 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:06:48,520 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:06:48,523 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:06:48,526 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:06:48,528 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:06:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:06:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:50.814677+00:00 (in 2.082045 seconds)
notificationworker stdout | 2025-02-07 04:06:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:58 UTC)" (scheduled at 2025-02-07 04:06:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:06:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:06:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 48, 732939), True, datetime.datetime(2025, 2, 7, 4, 6, 48, 732939), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:06:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:06:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:06:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:06:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:06:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:06:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:06:58.732147+00:00 (in 7.916959 seconds)
notificationworker stdout | 2025-02-07 04:06:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:50 UTC)" (scheduled at 2025-02-07 04:06:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:06:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:06:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:06:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:06:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:10.424250+00:00 (in 16.000739 seconds)
gcworker stdout | 2025-02-07 04:06:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:24 UTC)" (scheduled at 2025-02-07 04:06:54.423055+00:00)
gcworker stdout | 2025-02-07 04:06:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:06:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:06:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:06:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:56.118855+00:00 (in 59.999486 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:06:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:56 UTC)" (scheduled at 2025-02-07 04:06:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:06:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:06:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:06:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:37.715878+00:00 (in 41.001876 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:56 UTC)" (scheduled at 2025-02-07 04:06:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:06:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:06:57,176 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:06:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:06:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:08.732147+00:00 (in 9.999493 seconds)
notificationworker stdout | 2025-02-07 04:06:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:08 UTC)" (scheduled at 2025-02-07 04:06:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:06:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:06:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 6, 58, 732941), True, datetime.datetime(2025, 2, 7, 4, 6, 58, 732941), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:06:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:06:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:06:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:06:59,087 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:06:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:06:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:14.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 04:06:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:14 UTC)" (scheduled at 2025-02-07 04:06:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:06:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:06:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:06:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:06:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:06:59,235 [236] [DEBUG] [app] Starting request: urn:request:0d843f33-bbcf-4bd4-b0e2-1d4a7d102e28 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:59,235 [235] [DEBUG] [app] Starting request: urn:request:1742fcc6-231a-43e9-a601-9bc7381089f7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:06:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:06:59,238 [239] [DEBUG] [app] Starting request: urn:request:0e94627f-e965-4ee2-9376-89f5872cb989 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:59,238 [244] [DEBUG] [app] Starting request: urn:request:7403f594-ad14-4358-94d7-0a2129e60d45 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:06:59,239 [239] [DEBUG] [app] Ending request: urn:request:0e94627f-e965-4ee2-9376-89f5872cb989 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e94627f-e965-4ee2-9376-89f5872cb989', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:06:59,239 [244] [DEBUG] [app] Ending request: urn:request:7403f594-ad14-4358-94d7-0a2129e60d45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7403f594-ad14-4358-94d7-0a2129e60d45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:06:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:06:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:06:59,242 [236] [DEBUG] [app] Starting request: urn:request:ba0fcc40-889c-496a-96fa-89e9dbb5ad33 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:06:59,242 [235] [DEBUG] [app] Starting request: urn:request:19c5d4a5-c426-4ca3-9fc0-95dc6ec1d0d5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:06:59,242 [235] [DEBUG] [app] Ending request: urn:request:19c5d4a5-c426-4ca3-9fc0-95dc6ec1d0d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:19c5d4a5-c426-4ca3-9fc0-95dc6ec1d0d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:06:59,242 [236] [DEBUG] [app] Ending request: urn:request:ba0fcc40-889c-496a-96fa-89e9dbb5ad33 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba0fcc40-889c-496a-96fa-89e9dbb5ad33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:06:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:06:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:06:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:06:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:06:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:06:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:06:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:06:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:06:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:06:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:06:59,254 [236] [DEBUG] [app] Ending request: urn:request:0d843f33-bbcf-4bd4-b0e2-1d4a7d102e28 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d843f33-bbcf-4bd4-b0e2-1d4a7d102e28', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:06:59,254 [235] [DEBUG] [app] Ending request: urn:request:1742fcc6-231a-43e9-a601-9bc7381089f7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1742fcc6-231a-43e9-a601-9bc7381089f7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:06:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:06:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:06:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:06:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:06:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:06:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:06:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:04.936688+00:00 (in 4.998340 seconds)
exportactionlogsworker stdout | 2025-02-07 04:06:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:59 UTC)" (scheduled at 2025-02-07 04:06:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:06:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:06:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:07:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:07:00,248 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:07:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:07:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:23.420886+00:00 (in 20.001311 seconds)
repositorygcworker stdout | 2025-02-07 04:07:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:03 UTC)" (scheduled at 2025-02-07 04:07:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:07:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:07:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 3, 419780), True, datetime.datetime(2025, 2, 7, 4, 7, 3, 419780), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:07:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:07:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:07:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:07:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:07:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:33.919286+00:00 (in 29.999546 seconds)
buildlogsarchiver stdout | 2025-02-07 04:07:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:33 UTC)" (scheduled at 2025-02-07 04:07:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:07:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 7, 3, 919993), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:07:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:07:03,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:07:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:07:04,520 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:07:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:07:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:59.937899+00:00 (in 55.000734 seconds)
exportactionlogsworker stdout | 2025-02-07 04:07:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:04 UTC)" (scheduled at 2025-02-07 04:07:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:07:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:07:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 4, 937465), True, datetime.datetime(2025, 2, 7, 4, 7, 4, 937465), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:07:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:07:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:07:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:07:05,530 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:07:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:18.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 04:07:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:18 UTC)" (scheduled at 2025-02-07 04:07:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:07:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:07:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 8, 732873), True, datetime.datetime(2025, 2, 7, 4, 7, 8, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:07:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:07:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:07:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:07:09,615 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:07:10,156 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:07:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:07:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:24.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 04:07:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:40 UTC)" (scheduled at 2025-02-07 04:07:10.424250+00:00)
gcworker stdout | 2025-02-07 04:07:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:07:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900930431, None, 1, 0])
gcworker stdout | 2025-02-07 04:07:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:07:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:07:10,661 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:07:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:07:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:41.831521+00:00 (in 29.999542 seconds)
autopruneworker stdout | 2025-02-07 04:07:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:41 UTC)" (scheduled at 2025-02-07 04:07:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:07:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897631836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:07:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:07:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:07:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:07:11,920 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:07:12,842 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:07:12,890 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
namespacegcworker stdout | 2025-02-07 04:07:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:07:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:31.621131+00:00 (in 17.996366 seconds)
namespacegcworker stdout | 2025-02-07 04:07:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:12:13 UTC)" (scheduled at 2025-02-07 04:07:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:07:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 7, 13, 625045), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:07:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 7, 13, 625045), True, datetime.datetime(2025, 2, 7, 4, 7, 13, 625045), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:07:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 7, 13, 625045), True, datetime.datetime(2025, 2, 7, 4, 7, 13, 625045), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 7, 13, 625045), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:07:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:07:13,636 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:12:13 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:07:13,989 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:07:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:07:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:29.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 04:07:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:29 UTC)" (scheduled at 2025-02-07 04:07:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:07:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:07:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:07:14,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:07:14,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:07:14,236 [235] [DEBUG] [app] Starting request: urn:request:0cd4128c-934b-4d85-a626-cb8d2da9e9d2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:14,236 [238] [DEBUG] [app] Starting request: urn:request:13cc2e69-2091-4c64-9509-750c29f11b1d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:07:14,240 [246] [DEBUG] [app] Starting request: urn:request:aa9a045e-0d8a-4ced-9398-b770dca215f5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:14,240 [245] [DEBUG] [app] Starting request: urn:request:aa8fb242-8b78-4066-bd9d-cd488e778e9b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:14,241 [245] [DEBUG] [app] Ending request: urn:request:aa8fb242-8b78-4066-bd9d-cd488e778e9b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa8fb242-8b78-4066-bd9d-cd488e778e9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:07:14,241 [246] [DEBUG] [app] Ending request: urn:request:aa9a045e-0d8a-4ced-9398-b770dca215f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa9a045e-0d8a-4ced-9398-b770dca215f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:07:14,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:07:14,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:07:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:14,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:14,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:14,245 [235] [DEBUG] [app] Starting request: urn:request:03baaa1b-e2d0-4052-9a36-957bb2ba7a2d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:07:14,246 [237] [DEBUG] [app] Starting request: urn:request:efb9f55d-36db-4e2f-9808-e5391aeac670 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:07:14,246 [235] [DEBUG] [app] Ending request: urn:request:03baaa1b-e2d0-4052-9a36-957bb2ba7a2d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03baaa1b-e2d0-4052-9a36-957bb2ba7a2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:07:14,246 [237] [DEBUG] [app] Ending request: urn:request:efb9f55d-36db-4e2f-9808-e5391aeac670 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:efb9f55d-36db-4e2f-9808-e5391aeac670', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:07:14,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:07:14,247 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:07:14,247 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:07:14,247 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:14,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:14,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:14,247 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:14,247 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:14,252 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:14,252 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:14,257 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:14,257 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:14,258 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:14,258 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:14,260 [238] [DEBUG] [app] Ending request: urn:request:13cc2e69-2091-4c64-9509-750c29f11b1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13cc2e69-2091-4c64-9509-750c29f11b1d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:14,260 [235] [DEBUG] [app] Ending request: urn:request:0cd4128c-934b-4d85-a626-cb8d2da9e9d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0cd4128c-934b-4d85-a626-cb8d2da9e9d2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:14,260 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:07:14,260 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:07:14,261 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:07:14,261 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.026 118 0.026)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.026 118 0.026)
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:14,329 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:07:14,996 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:07:15,040 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:07:15,135 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:07:15,542 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:07:15,901 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:07:16,064 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:07:16,121 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:07:16,394 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:07:16,398 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:07:16,400 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:07:16,403 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:07:16,405 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:07:17,415 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:07:17,418 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:07:17,421 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:07:17,738 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:07:17,956 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:07:18,180 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:07:18,508 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:07:18,514 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:07:18,516 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:07:18,519 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:07:18,522 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:07:18,529 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:07:18,533 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:07:18,535 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:07:18,538 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:07:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:28.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 04:07:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:28 UTC)" (scheduled at 2025-02-07 04:07:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:07:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:07:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 18, 732878), True, datetime.datetime(2025, 2, 7, 4, 7, 18, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:07:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:07:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:07:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:07:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:07:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:03.419110+00:00 (in 39.997724 seconds)
repositorygcworker stdout | 2025-02-07 04:07:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:23 UTC)" (scheduled at 2025-02-07 04:07:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:07:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:07:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:07:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:07:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:40.424250+00:00 (in 16.000741 seconds)
gcworker stdout | 2025-02-07 04:07:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:54 UTC)" (scheduled at 2025-02-07 04:07:24.423055+00:00)
gcworker stdout | 2025-02-07 04:07:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:07:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:07:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:07:27,207 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:07:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:33.814166+00:00 (in 5.081568 seconds)
notificationworker stdout | 2025-02-07 04:07:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:38 UTC)" (scheduled at 2025-02-07 04:07:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:07:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:07:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 28, 732804), True, datetime.datetime(2025, 2, 7, 4, 7, 28, 732804), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:07:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:07:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:07:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:07:29,124 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:07:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:07:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:44.225378+00:00 (in 14.999515 seconds)
quotatotalworker stdout | 2025-02-07 04:07:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:44 UTC)" (scheduled at 2025-02-07 04:07:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:07:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:07:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:07:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:07:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:07:29,235 [236] [DEBUG] [app] Starting request: urn:request:e375af95-06e9-43c4-a639-fdbd47ab1ddb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:29,235 [235] [DEBUG] [app] Starting request: urn:request:d7b69b8f-06e9-41d5-b032-28acb3fa1f6a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:07:29,239 [239] [DEBUG] [app] Starting request: urn:request:ddb3e2ba-47bc-4aea-9df0-5206f1624a95 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:29,239 [246] [DEBUG] [app] Starting request: urn:request:b8e1eeca-ca96-444f-9172-68f3164bfb3b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:29,239 [239] [DEBUG] [app] Ending request: urn:request:ddb3e2ba-47bc-4aea-9df0-5206f1624a95 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ddb3e2ba-47bc-4aea-9df0-5206f1624a95', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:07:29,239 [246] [DEBUG] [app] Ending request: urn:request:b8e1eeca-ca96-444f-9172-68f3164bfb3b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b8e1eeca-ca96-444f-9172-68f3164bfb3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:07:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:07:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:07:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:29,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:29,243 [238] [DEBUG] [app] Starting request: urn:request:439bd86f-99b6-42e3-8458-7660a704f026 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:07:29,243 [238] [DEBUG] [app] Ending request: urn:request:439bd86f-99b6-42e3-8458-7660a704f026 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:439bd86f-99b6-42e3-8458-7660a704f026', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:07:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:07:29,243 [235] [DEBUG] [app] Starting request: urn:request:bb4f3c8d-4fde-425b-bb49-1b45130d45e3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:07:29,243 [235] [DEBUG] [app] Ending request: urn:request:bb4f3c8d-4fde-425b-bb49-1b45130d45e3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb4f3c8d-4fde-425b-bb49-1b45130d45e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:07:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:07:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:29,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:29,255 [235] [DEBUG] [app] Ending request: urn:request:d7b69b8f-06e9-41d5-b032-28acb3fa1f6a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d7b69b8f-06e9-41d5-b032-28acb3fa1f6a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:07:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
gunicorn-web stdout | 2025-02-07 04:07:29,255 [236] [DEBUG] [app] Ending request: urn:request:e375af95-06e9-43c4-a639-fdbd47ab1ddb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e375af95-06e9-43c4-a639-fdbd47ab1ddb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:29,256 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
gunicorn-web stdout | 2025-02-07 04:07:29,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 04:07:30,284 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:07:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:07:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:32.625945+00:00 (in 1.004382 seconds)
namespacegcworker stdout | 2025-02-07 04:07:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:31 UTC)" (scheduled at 2025-02-07 04:07:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:07:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:07:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 31, 621841), True, datetime.datetime(2025, 2, 7, 4, 7, 31, 621841), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:07:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:07:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:07:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:07:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:07:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:31.621131+00:00 (in 58.994713 seconds)
namespacegcworker stdout | 2025-02-07 04:07:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:32 UTC)" (scheduled at 2025-02-07 04:07:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:07:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:07:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:07:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:38.732147+00:00 (in 4.917499 seconds)
notificationworker stdout | 2025-02-07 04:07:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:12:33 UTC)" (scheduled at 2025-02-07 04:07:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:07:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 7, 33, 814832), 'notification/%'])
notificationworker stdout | 2025-02-07 04:07:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 7, 33, 814832), True, datetime.datetime(2025, 2, 7, 4, 7, 33, 814832), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:07:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 7, 33, 814832), True, datetime.datetime(2025, 2, 7, 4, 7, 33, 814832), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 7, 33, 814832), 'notification/%'])
notificationworker stdout | 2025-02-07 04:07:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:07:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:12:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:07:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:07:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:03.919286+00:00 (in 29.999540 seconds)
buildlogsarchiver stdout | 2025-02-07 04:07:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:03 UTC)" (scheduled at 2025-02-07 04:07:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:07:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 7, 33, 920050), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:07:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:07:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:07:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:07:34,527 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:07:35,567 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:56.713547+00:00 (in 18.997180 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:37 UTC)" (scheduled at 2025-02-07 04:07:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:07:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:48.732147+00:00 (in 9.999516 seconds)
notificationworker stdout | 2025-02-07 04:07:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:48 UTC)" (scheduled at 2025-02-07 04:07:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:07:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:07:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 38, 732840), True, datetime.datetime(2025, 2, 7, 4, 7, 38, 732840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:07:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:07:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:07:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:07:39,652 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:07:40,178 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:07:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:07:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:54.423055+00:00 (in 13.998339 seconds)
gcworker stdout | 2025-02-07 04:07:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:10 UTC)" (scheduled at 2025-02-07 04:07:40.424250+00:00)
gcworker stdout | 2025-02-07 04:07:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:07:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900960431, None, 1, 0])
gcworker stdout | 2025-02-07 04:07:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:07:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:07:40,697 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:07:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:07:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:11.831521+00:00 (in 29.999559 seconds)
autopruneworker stdout | 2025-02-07 04:07:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:11 UTC)" (scheduled at 2025-02-07 04:07:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:07:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897661836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:07:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:07:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:07:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:07:41,951 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:07:42,854 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:07:42,927 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:07:44,025 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:07:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:07:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:59.225378+00:00 (in 14.999481 seconds)
quotatotalworker stdout | 2025-02-07 04:07:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:59 UTC)" (scheduled at 2025-02-07 04:07:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:07:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:07:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:07:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:07:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:07:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:07:44,235 [236] [DEBUG] [app] Starting request: urn:request:07d5a472-69df-45a7-8ec3-224955cfd951 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:44,235 [235] [DEBUG] [app] Starting request: urn:request:c762b498-5b98-4f79-bcca-52807469d4ba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:07:44,239 [239] [DEBUG] [app] Starting request: urn:request:7343802b-d992-424e-ad53-6813ab7dce7f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:44,239 [245] [DEBUG] [app] Starting request: urn:request:1eba8689-46b8-420a-87c4-d417e1101540 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:44,239 [239] [DEBUG] [app] Ending request: urn:request:7343802b-d992-424e-ad53-6813ab7dce7f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7343802b-d992-424e-ad53-6813ab7dce7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:07:44,239 [245] [DEBUG] [app] Ending request: urn:request:1eba8689-46b8-420a-87c4-d417e1101540 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1eba8689-46b8-420a-87c4-d417e1101540', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:07:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:07:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:07:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:44,242 [235] [DEBUG] [app] Starting request: urn:request:e951902e-3c5c-4448-930f-0934dd11bc0d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:07:44,242 [238] [DEBUG] [app] Starting request: urn:request:98e0328e-8c1c-412d-957d-bd3683ca87df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:07:44,242 [235] [DEBUG] [app] Ending request: urn:request:e951902e-3c5c-4448-930f-0934dd11bc0d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e951902e-3c5c-4448-930f-0934dd11bc0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:07:44,243 [238] [DEBUG] [app] Ending request: urn:request:98e0328e-8c1c-412d-957d-bd3683ca87df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98e0328e-8c1c-412d-957d-bd3683ca87df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:07:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:07:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:07:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:44,254 [235] [DEBUG] [app] Ending request: urn:request:c762b498-5b98-4f79-bcca-52807469d4ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c762b498-5b98-4f79-bcca-52807469d4ba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:44,254 [236] [DEBUG] [app] Ending request: urn:request:07d5a472-69df-45a7-8ec3-224955cfd951 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07d5a472-69df-45a7-8ec3-224955cfd951', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:07:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:07:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:07:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:44,360 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:07:45,003 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:07:45,077 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:07:45,171 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:07:45,578 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:07:45,935 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:07:46,101 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:07:46,143 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:07:46,402 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:07:46,406 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:07:46,410 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:07:46,412 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:07:46,415 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:07:47,424 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:07:47,427 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:07:47,429 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:07:47,774 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:07:47,968 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:07:48,211 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:07:48,520 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:07:48,524 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:07:48,526 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:07:48,529 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:07:48,531 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:07:48,537 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:07:48,541 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:07:48,543 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:07:48,546 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:07:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:50.814677+00:00 (in 2.082086 seconds)
notificationworker stdout | 2025-02-07 04:07:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:58 UTC)" (scheduled at 2025-02-07 04:07:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:07:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:07:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 48, 732801), True, datetime.datetime(2025, 2, 7, 4, 7, 48, 732801), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:07:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:07:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:07:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:07:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:07:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:07:58.732147+00:00 (in 7.916973 seconds)
notificationworker stdout | 2025-02-07 04:07:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:50 UTC)" (scheduled at 2025-02-07 04:07:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:07:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:07:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:07:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:07:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:10.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 04:07:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:24 UTC)" (scheduled at 2025-02-07 04:07:54.423055+00:00)
gcworker stdout | 2025-02-07 04:07:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:07:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:07:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:07:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:56.118855+00:00 (in 59.999469 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:07:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:56 UTC)" (scheduled at 2025-02-07 04:07:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:07:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:07:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:07:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:37.715878+00:00 (in 41.001876 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:56 UTC)" (scheduled at 2025-02-07 04:07:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:07:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:07:57,235 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:07:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:07:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:08.732147+00:00 (in 9.999499 seconds)
notificationworker stdout | 2025-02-07 04:07:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:08 UTC)" (scheduled at 2025-02-07 04:07:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:07:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:07:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 7, 58, 732940), True, datetime.datetime(2025, 2, 7, 4, 7, 58, 732940), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:07:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:07:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:07:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:07:59,141 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:07:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:07:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:14.225378+00:00 (in 14.999525 seconds)
quotatotalworker stdout | 2025-02-07 04:07:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:14 UTC)" (scheduled at 2025-02-07 04:07:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:07:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:07:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:07:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:07:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:07:59,236 [238] [DEBUG] [app] Starting request: urn:request:3af1a99d-8d3d-4438-a745-0b81f65010d2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:59,236 [237] [DEBUG] [app] Starting request: urn:request:f5bc5aa4-3684-4205-9a6a-a2f4f50046b8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:07:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:07:59,239 [239] [DEBUG] [app] Starting request: urn:request:cda67f25-b70e-4a78-bc73-5684eae8d476 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:59,239 [246] [DEBUG] [app] Starting request: urn:request:14cd6c2e-a4c5-41a1-b66d-2c06f8cebbf5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:07:59,239 [239] [DEBUG] [app] Ending request: urn:request:cda67f25-b70e-4a78-bc73-5684eae8d476 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cda67f25-b70e-4a78-bc73-5684eae8d476', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:07:59,239 [246] [DEBUG] [app] Ending request: urn:request:14cd6c2e-a4c5-41a1-b66d-2c06f8cebbf5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:14cd6c2e-a4c5-41a1-b66d-2c06f8cebbf5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:07:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:07:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:07:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:07:59,242 [237] [DEBUG] [app] Starting request: urn:request:29cecc36-f708-4c8e-b0f9-b9b85b58cbd2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:07:59,242 [237] [DEBUG] [app] Ending request: urn:request:29cecc36-f708-4c8e-b0f9-b9b85b58cbd2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:29cecc36-f708-4c8e-b0f9-b9b85b58cbd2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:07:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:07:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:59,243 [236] [DEBUG] [app] Starting request: urn:request:7cea262f-8bf3-4dca-9a87-10e910dfbf97 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:07:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:59,243 [236] [DEBUG] [app] Ending request: urn:request:7cea262f-8bf3-4dca-9a87-10e910dfbf97 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7cea262f-8bf3-4dca-9a87-10e910dfbf97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:07:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:07:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:07:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:07:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:07:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:07:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:07:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:07:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:07:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:07:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:07:59,254 [237] [DEBUG] [app] Ending request: urn:request:f5bc5aa4-3684-4205-9a6a-a2f4f50046b8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f5bc5aa4-3684-4205-9a6a-a2f4f50046b8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:07:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:07:59,255 [238] [DEBUG] [app] Ending request: urn:request:3af1a99d-8d3d-4438-a745-0b81f65010d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3af1a99d-8d3d-4438-a745-0b81f65010d2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:07:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:07:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 04:07:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:07:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:07:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:07:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:04.936688+00:00 (in 4.998305 seconds)
exportactionlogsworker stdout | 2025-02-07 04:07:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:59 UTC)" (scheduled at 2025-02-07 04:07:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:07:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:07:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:08:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:08:00,291 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:08:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:08:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:23.420886+00:00 (in 20.001301 seconds)
repositorygcworker stdout | 2025-02-07 04:08:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:03 UTC)" (scheduled at 2025-02-07 04:08:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:08:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:08:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 3, 419862), True, datetime.datetime(2025, 2, 7, 4, 8, 3, 419862), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:08:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:08:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:08:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:08:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:08:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:33.919286+00:00 (in 29.999532 seconds)
buildlogsarchiver stdout | 2025-02-07 04:08:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:33 UTC)" (scheduled at 2025-02-07 04:08:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:08:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 8, 3, 920004), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:08:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:08:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:08:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:08:04,563 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:08:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:08:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:59.937899+00:00 (in 55.000738 seconds)
exportactionlogsworker stdout | 2025-02-07 04:08:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:04 UTC)" (scheduled at 2025-02-07 04:08:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:08:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:08:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 4, 937471), True, datetime.datetime(2025, 2, 7, 4, 8, 4, 937471), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:08:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:08:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:08:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:08:05,601 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:08:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:08:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:18.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 04:08:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:18 UTC)" (scheduled at 2025-02-07 04:08:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:08:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:08:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 8, 732854), True, datetime.datetime(2025, 2, 7, 4, 8, 8, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:08:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:08:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:08:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:08:09,667 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:08:10,215 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:08:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:08:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:24.423055+00:00 (in 13.998360 seconds)
gcworker stdout | 2025-02-07 04:08:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:40 UTC)" (scheduled at 2025-02-07 04:08:10.424250+00:00)
gcworker stdout | 2025-02-07 04:08:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:08:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738900990431, None, 1, 0])
gcworker stdout | 2025-02-07 04:08:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:08:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:08:10,731 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:08:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:08:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:41.831521+00:00 (in 29.999539 seconds)
autopruneworker stdout | 2025-02-07 04:08:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:41 UTC)" (scheduled at 2025-02-07 04:08:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:08:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897691836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:08:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:08:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:08:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:08:11,987 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:08:12,890 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:08:12,963 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:08:14,042 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:08:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:08:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:29.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 04:08:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:29 UTC)" (scheduled at 2025-02-07 04:08:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:08:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:08:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:08:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:08:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:08:14,236 [235] [DEBUG] [app] Starting request: urn:request:dcf1f1da-1278-4c62-ab65-52d758879972 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:14,236 [236] [DEBUG] [app] Starting request: urn:request:c4443b60-4827-4ee8-aa46-493c8bc19ac6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:14,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:08:14,240 [239] [DEBUG] [app] Starting request: urn:request:c65caaa4-65b0-4dc8-b54d-6e4c0fd4b1ca (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:08:14,240 [246] [DEBUG] [app] Starting request: urn:request:8a62feea-0a86-42ca-8f22-beea3d08d35c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:08:14,240 [239] [DEBUG] [app] Ending request: urn:request:c65caaa4-65b0-4dc8-b54d-6e4c0fd4b1ca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c65caaa4-65b0-4dc8-b54d-6e4c0fd4b1ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:08:14,240 [246] [DEBUG] [app] Ending request: urn:request:8a62feea-0a86-42ca-8f22-beea3d08d35c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a62feea-0a86-42ca-8f22-beea3d08d35c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:08:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:08:14,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:14,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:14,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:08:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:14,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:14,244 [237] [DEBUG] [app] Starting request: urn:request:35ad45fe-b9ac-4a25-8f21-b38c4488474a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:14,244 [238] [DEBUG] [app] Starting request: urn:request:a11390a8-14bb-4f94-9816-c8e388da5d60 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:14,244 [238] [DEBUG] [app] Ending request: urn:request:a11390a8-14bb-4f94-9816-c8e388da5d60 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a11390a8-14bb-4f94-9816-c8e388da5d60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:08:14,244 [237] [DEBUG] [app] Ending request: urn:request:35ad45fe-b9ac-4a25-8f21-b38c4488474a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:35ad45fe-b9ac-4a25-8f21-b38c4488474a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:08:14,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:14,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:14,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:14,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:14,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:14,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:14,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:14,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:14,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:14,256 [236] [DEBUG] [app] Ending request: urn:request:c4443b60-4827-4ee8-aa46-493c8bc19ac6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4443b60-4827-4ee8-aa46-493c8bc19ac6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:14,256 [235] [DEBUG] [app] Ending request: urn:request:dcf1f1da-1278-4c62-ab65-52d758879972 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcf1f1da-1278-4c62-ab65-52d758879972', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:14,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:08:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:08:14,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:08:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:14,396 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:08:15,031 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:08:15,098 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:08:15,207 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:08:15,604 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:08:15,971 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:08:16,138 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:08:16,180 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:08:16,411 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:08:16,414 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:08:16,417 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:08:16,419 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:08:16,422 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:08:17,432 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:08:17,436 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:08:17,438 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:08:17,810 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:08:18,004 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:08:18,247 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:08:18,531 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:08:18,535 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:08:18,538 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:08:18,541 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:08:18,543 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:08:18,546 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:08:18,548 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:08:18,552 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:08:18,555 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:08:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:08:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:28.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 04:08:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:28 UTC)" (scheduled at 2025-02-07 04:08:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:08:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:08:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 18, 732879), True, datetime.datetime(2025, 2, 7, 4, 8, 18, 732879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:08:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:08:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:08:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:08:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:08:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:32.420707+00:00 (in 8.999358 seconds)
repositorygcworker stdout | 2025-02-07 04:08:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:23 UTC)" (scheduled at 2025-02-07 04:08:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:08:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:08:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:08:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:08:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:40.424250+00:00 (in 16.000744 seconds)
gcworker stdout | 2025-02-07 04:08:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:54 UTC)" (scheduled at 2025-02-07 04:08:24.423055+00:00)
gcworker stdout | 2025-02-07 04:08:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:08:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:08:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:08:27,248 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:08:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:08:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:38.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 04:08:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:38 UTC)" (scheduled at 2025-02-07 04:08:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:08:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:08:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 28, 732900), True, datetime.datetime(2025, 2, 7, 4, 8, 28, 732900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:08:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:08:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:08:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:38 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:08:29,177 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:08:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:08:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:44.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 04:08:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:44 UTC)" (scheduled at 2025-02-07 04:08:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:08:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:08:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:08:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:08:29,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:08:29,236 [238] [DEBUG] [app] Starting request: urn:request:390f49b0-838a-4697-bd7c-3607f8677f70 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:29,236 [235] [DEBUG] [app] Starting request: urn:request:0c9d03c8-d42f-49b3-b04f-0b98b26cf33e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:08:29,239 [239] [DEBUG] [app] Starting request: urn:request:21478bf0-4811-4089-9b41-2a21874b3688 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:08:29,239 [239] [DEBUG] [app] Ending request: urn:request:21478bf0-4811-4089-9b41-2a21874b3688 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:21478bf0-4811-4089-9b41-2a21874b3688', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:08:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:08:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:08:29,240 [242] [DEBUG] [app] Starting request: urn:request:6fd9c489-362e-410a-97a4-3533bbe2956f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:08:29,241 [242] [DEBUG] [app] Ending request: urn:request:6fd9c489-362e-410a-97a4-3533bbe2956f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6fd9c489-362e-410a-97a4-3533bbe2956f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.004 162 0.004)
gunicorn-registry stdout | 2025-02-07 04:08:29,242 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:29,242 [238] [DEBUG] [app] Starting request: urn:request:79c7e827-9329-4772-a5b0-94c2e2ef9ff2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:29,242 [238] [DEBUG] [app] Ending request: urn:request:79c7e827-9329-4772-a5b0-94c2e2ef9ff2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79c7e827-9329-4772-a5b0-94c2e2ef9ff2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:08:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:08:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:29,244 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:29,246 [237] [DEBUG] [app] Starting request: urn:request:44667d50-8145-4a60-9ccf-49ccedbe518e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:29,246 [237] [DEBUG] [app] Ending request: urn:request:44667d50-8145-4a60-9ccf-49ccedbe518e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:44667d50-8145-4a60-9ccf-49ccedbe518e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:08:29,247 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:29,247 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:29,247 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:29,247 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:29,251 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:29,255 [235] [DEBUG] [app] Ending request: urn:request:0c9d03c8-d42f-49b3-b04f-0b98b26cf33e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c9d03c8-d42f-49b3-b04f-0b98b26cf33e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:08:29,256 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:08:29,257 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:29,258 [238] [DEBUG] [app] Ending request: urn:request:390f49b0-838a-4697-bd7c-3607f8677f70 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:390f49b0-838a-4697-bd7c-3607f8677f70', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:29,258 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gunicorn-web stdout | 2025-02-07 04:08:29,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
queuecleanupworker stdout | 2025-02-07 04:08:30,327 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:08:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:08:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:32.625945+00:00 (in 1.004279 seconds)
namespacegcworker stdout | 2025-02-07 04:08:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:31 UTC)" (scheduled at 2025-02-07 04:08:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:08:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:08:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 31, 621991), True, datetime.datetime(2025, 2, 7, 4, 8, 31, 621991), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:08:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:08:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:08:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:08:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:08:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:03.419110+00:00 (in 30.997909 seconds)
repositorygcworker stdout | 2025-02-07 04:08:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:13:32 UTC)" (scheduled at 2025-02-07 04:08:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:08:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 8, 32, 421439), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:08:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 8, 32, 421439), True, datetime.datetime(2025, 2, 7, 4, 8, 32, 421439), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:08:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 8, 32, 421439), True, datetime.datetime(2025, 2, 7, 4, 8, 32, 421439), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 8, 32, 421439), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:08:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:08:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:13:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:08:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:08:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:31.621131+00:00 (in 58.994663 seconds)
namespacegcworker stdout | 2025-02-07 04:08:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:32 UTC)" (scheduled at 2025-02-07 04:08:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:08:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:08:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:08:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:08:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:03.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 04:08:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:03 UTC)" (scheduled at 2025-02-07 04:08:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:08:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 8, 33, 919998), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:08:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:08:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:08:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:08:34,578 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:08:35,618 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:56.713547+00:00 (in 18.997206 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:37 UTC)" (scheduled at 2025-02-07 04:08:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:08:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:08:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:48.732147+00:00 (in 9.999416 seconds)
notificationworker stdout | 2025-02-07 04:08:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:48 UTC)" (scheduled at 2025-02-07 04:08:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:08:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:08:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 38, 733109), True, datetime.datetime(2025, 2, 7, 4, 8, 38, 733109), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:08:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:08:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:08:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:08:39,703 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:08:40,246 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:08:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:08:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:54.423055+00:00 (in 13.998371 seconds)
gcworker stdout | 2025-02-07 04:08:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:10 UTC)" (scheduled at 2025-02-07 04:08:40.424250+00:00)
gcworker stdout | 2025-02-07 04:08:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:08:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901020431, None, 1, 0])
gcworker stdout | 2025-02-07 04:08:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:08:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:08:40,764 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:08:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:08:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:11.831521+00:00 (in 29.999524 seconds)
autopruneworker stdout | 2025-02-07 04:08:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:11 UTC)" (scheduled at 2025-02-07 04:08:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:08:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897721836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:08:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:08:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:08:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:08:42,023 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:08:42,925 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:08:42,999 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:08:44,078 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:08:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:08:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:59.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 04:08:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:59 UTC)" (scheduled at 2025-02-07 04:08:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:08:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:08:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:08:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:08:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:08:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:08:44,235 [238] [DEBUG] [app] Starting request: urn:request:0bf784aa-dee1-4ba0-82fd-e8231b5d9ad9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:44,235 [235] [DEBUG] [app] Starting request: urn:request:9febe10d-a5d5-4825-8cde-c35cc2e70ca6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:08:44,238 [239] [DEBUG] [app] Starting request: urn:request:4bfa2027-3998-4c92-8d82-039908b8261a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:08:44,238 [244] [DEBUG] [app] Starting request: urn:request:5c0407ff-8662-4fda-8612-66ac28d657d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:08:44,238 [244] [DEBUG] [app] Ending request: urn:request:5c0407ff-8662-4fda-8612-66ac28d657d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c0407ff-8662-4fda-8612-66ac28d657d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:08:44,238 [239] [DEBUG] [app] Ending request: urn:request:4bfa2027-3998-4c92-8d82-039908b8261a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4bfa2027-3998-4c92-8d82-039908b8261a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:08:44,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:08:44,238 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:44,238 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:44,238 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:44,241 [238] [DEBUG] [app] Starting request: urn:request:d4f1c810-725b-4fa1-8c55-283988f56acd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:44,241 [238] [DEBUG] [app] Ending request: urn:request:d4f1c810-725b-4fa1-8c55-283988f56acd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4f1c810-725b-4fa1-8c55-283988f56acd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:08:44,241 [237] [DEBUG] [app] Starting request: urn:request:32220a8e-f549-4666-9255-a99b3e3130b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:44,241 [237] [DEBUG] [app] Ending request: urn:request:32220a8e-f549-4666-9255-a99b3e3130b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32220a8e-f549-4666-9255-a99b3e3130b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:08:44,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:08:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:08:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:44,245 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:44,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:44,252 [238] [DEBUG] [app] Ending request: urn:request:0bf784aa-dee1-4ba0-82fd-e8231b5d9ad9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0bf784aa-dee1-4ba0-82fd-e8231b5d9ad9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:44,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:08:44,252 [235] [DEBUG] [app] Ending request: urn:request:9febe10d-a5d5-4825-8cde-c35cc2e70ca6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9febe10d-a5d5-4825-8cde-c35cc2e70ca6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:44,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:08:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
gunicorn-web stdout | 2025-02-07 04:08:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:44,432 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:08:45,058 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:08:45,134 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:08:45,228 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:08:45,626 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:08:46,007 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:08:46,148 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:08:46,194 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:08:46,420 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:08:46,423 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:08:46,426 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:08:46,429 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:08:46,431 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:08:47,441 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:08:47,444 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:08:47,446 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:08:47,818 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:08:48,018 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:08:48,265 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:08:48,542 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:08:48,545 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:08:48,547 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:08:48,550 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:08:48,554 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:08:48,556 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:08:48,559 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:08:48,562 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:08:48,565 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:08:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:08:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:50.814677+00:00 (in 2.082027 seconds)
notificationworker stdout | 2025-02-07 04:08:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:58 UTC)" (scheduled at 2025-02-07 04:08:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:08:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:08:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 48, 732956), True, datetime.datetime(2025, 2, 7, 4, 8, 48, 732956), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:08:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:08:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:08:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:08:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:08:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:08:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:08:58.732147+00:00 (in 7.916940 seconds)
notificationworker stdout | 2025-02-07 04:08:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:50 UTC)" (scheduled at 2025-02-07 04:08:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:08:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:08:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:08:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:08:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:10.424250+00:00 (in 16.000748 seconds)
gcworker stdout | 2025-02-07 04:08:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:24 UTC)" (scheduled at 2025-02-07 04:08:54.423055+00:00)
gcworker stdout | 2025-02-07 04:08:54,423 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 04:08:54,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:08:54,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:08:54,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:08:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:08:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:56.118855+00:00 (in 59.999493 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:08:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:56 UTC)" (scheduled at 2025-02-07 04:08:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:08:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:08:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:08:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:37.715878+00:00 (in 41.001889 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:56 UTC)" (scheduled at 2025-02-07 04:08:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:08:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:08:57,280 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:08:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:08:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:08.732147+00:00 (in 9.999477 seconds)
notificationworker stdout | 2025-02-07 04:08:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:08 UTC)" (scheduled at 2025-02-07 04:08:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:08:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:08:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 8, 58, 732997), True, datetime.datetime(2025, 2, 7, 4, 8, 58, 732997), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:08:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:08:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:08:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:08 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:08:59,213 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
quotatotalworker stdout | 2025-02-07 04:08:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:08:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:14.225378+00:00 (in 14.999591 seconds)
quotatotalworker stdout | 2025-02-07 04:08:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:14 UTC)" (scheduled at 2025-02-07 04:08:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:08:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:08:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:08:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:08:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:08:59,236 [238] [DEBUG] [app] Starting request: urn:request:45f94e7f-3c13-42d7-9188-d81da448d8a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:59,236 [235] [DEBUG] [app] Starting request: urn:request:b7396127-ad93-4e4f-ad4b-79d3e6469859 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:08:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:08:59,239 [245] [DEBUG] [app] Starting request: urn:request:4ac52779-e180-418b-8938-df5e9d25e9f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:08:59,239 [244] [DEBUG] [app] Starting request: urn:request:7cc4a976-7ece-4b53-8437-be9c6f0ff793 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:08:59,239 [245] [DEBUG] [app] Ending request: urn:request:4ac52779-e180-418b-8938-df5e9d25e9f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ac52779-e180-418b-8938-df5e9d25e9f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:08:59,239 [244] [DEBUG] [app] Ending request: urn:request:7cc4a976-7ece-4b53-8437-be9c6f0ff793 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7cc4a976-7ece-4b53-8437-be9c6f0ff793', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:08:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:08:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:08:59,242 [238] [DEBUG] [app] Starting request: urn:request:92efd0ee-9ae3-46dc-9a1e-500de3786e6f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:59,242 [238] [DEBUG] [app] Ending request: urn:request:92efd0ee-9ae3-46dc-9a1e-500de3786e6f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92efd0ee-9ae3-46dc-9a1e-500de3786e6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:08:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:08:59,243 [236] [DEBUG] [app] Starting request: urn:request:5e5f5ca0-0fd0-4fb4-9e43-405c4f2db1c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:08:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:59,243 [236] [DEBUG] [app] Ending request: urn:request:5e5f5ca0-0fd0-4fb4-9e43-405c4f2db1c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5e5f5ca0-0fd0-4fb4-9e43-405c4f2db1c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:08:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:08:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:08:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:08:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:08:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:08:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:08:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:08:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:08:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:08:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:08:59,253 [238] [DEBUG] [app] Ending request: urn:request:45f94e7f-3c13-42d7-9188-d81da448d8a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:45f94e7f-3c13-42d7-9188-d81da448d8a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:08:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:08:59,255 [235] [DEBUG] [app] Ending request: urn:request:b7396127-ad93-4e4f-ad4b-79d3e6469859 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b7396127-ad93-4e4f-ad4b-79d3e6469859', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:08:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:08:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:08:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:08:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:08:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:08:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:04.936688+00:00 (in 4.998310 seconds)
exportactionlogsworker stdout | 2025-02-07 04:08:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:59 UTC)" (scheduled at 2025-02-07 04:08:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:08:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:08:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:09:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:09:00,363 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:09:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:09:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:23.420886+00:00 (in 20.001264 seconds)
repositorygcworker stdout | 2025-02-07 04:09:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:03 UTC)" (scheduled at 2025-02-07 04:09:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:09:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:09:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 3, 419838), True, datetime.datetime(2025, 2, 7, 4, 9, 3, 419838), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:09:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:09:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:09:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:09:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:09:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:33.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 04:09:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:33 UTC)" (scheduled at 2025-02-07 04:09:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:09:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 9, 3, 919988), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:09:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:09:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:09:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:09:04,614 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:09:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:09:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:52.937648+00:00 (in 48.000540 seconds)
exportactionlogsworker stdout | 2025-02-07 04:09:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:04 UTC)" (scheduled at 2025-02-07 04:09:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:09:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:09:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 4, 937407), True, datetime.datetime(2025, 2, 7, 4, 9, 4, 937407), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:09:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:09:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:09:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:09:05,654 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:09:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:09:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:18.732147+00:00 (in 9.999356 seconds)
notificationworker stdout | 2025-02-07 04:09:08,733 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:18 UTC)" (scheduled at 2025-02-07 04:09:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:09:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:09:08,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 8, 733292), True, datetime.datetime(2025, 2, 7, 4, 9, 8, 733292), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:09:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:09:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:09:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:09:09,735 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:09:10,267 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:09:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:09:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:24.423055+00:00 (in 13.998369 seconds)
gcworker stdout | 2025-02-07 04:09:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:40 UTC)" (scheduled at 2025-02-07 04:09:10.424250+00:00)
gcworker stdout | 2025-02-07 04:09:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:09:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901050431, None, 1, 0])
gcworker stdout | 2025-02-07 04:09:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:09:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:09:10,800 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:09:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:09:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 04:09:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:41 UTC)" (scheduled at 2025-02-07 04:09:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:09:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897751836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:09:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:09:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:09:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:09:12,058 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:09:12,957 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:09:13,025 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:09:14,114 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:09:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:09:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:29.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 04:09:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:29 UTC)" (scheduled at 2025-02-07 04:09:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:09:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:09:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:09:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:09:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:09:14,236 [237] [DEBUG] [app] Starting request: urn:request:03c1a0e1-90cf-4852-9b41-817bd9219059 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:14,236 [235] [DEBUG] [app] Starting request: urn:request:c3757858-0386-413c-aff7-30f332751578 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:09:14,239 [245] [DEBUG] [app] Starting request: urn:request:9fcb7eef-6d2c-49ca-9c55-aedfa8856420 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:14,239 [246] [DEBUG] [app] Starting request: urn:request:c1812164-11ae-4c20-940f-7bbc48ff8b8c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:14,239 [245] [DEBUG] [app] Ending request: urn:request:9fcb7eef-6d2c-49ca-9c55-aedfa8856420 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9fcb7eef-6d2c-49ca-9c55-aedfa8856420', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:09:14,239 [246] [DEBUG] [app] Ending request: urn:request:c1812164-11ae-4c20-940f-7bbc48ff8b8c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c1812164-11ae-4c20-940f-7bbc48ff8b8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:09:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:09:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:09:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:14,242 [237] [DEBUG] [app] Starting request: urn:request:9aa754cf-ce73-45a7-a981-0ad47931e294 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:14,242 [235] [DEBUG] [app] Starting request: urn:request:2a02506b-f01b-4e9a-9401-4fc94391358d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:14,242 [237] [DEBUG] [app] Ending request: urn:request:9aa754cf-ce73-45a7-a981-0ad47931e294 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9aa754cf-ce73-45a7-a981-0ad47931e294', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:09:14,242 [235] [DEBUG] [app] Ending request: urn:request:2a02506b-f01b-4e9a-9401-4fc94391358d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2a02506b-f01b-4e9a-9401-4fc94391358d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:09:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:09:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:09:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:14,254 [237] [DEBUG] [app] Ending request: urn:request:03c1a0e1-90cf-4852-9b41-817bd9219059 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:03c1a0e1-90cf-4852-9b41-817bd9219059', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:09:14,254 [235] [DEBUG] [app] Ending request: urn:request:c3757858-0386-413c-aff7-30f332751578 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3757858-0386-413c-aff7-30f332751578', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:09:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:09:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:09:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:09:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:14,470 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:09:15,095 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:09:15,170 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:09:15,264 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:09:15,662 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:09:16,043 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:09:16,185 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:09:16,230 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:09:16,427 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:09:16,430 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:09:16,434 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:09:16,437 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:09:16,440 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:09:17,450 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:09:17,453 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:09:17,455 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:09:17,855 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:09:18,054 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:09:18,301 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:09:18,552 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:09:18,555 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:09:18,559 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:09:18,563 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:09:18,566 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:09:18,569 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:09:18,572 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:09:18,574 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:09:18,577 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:09:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:09:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:28.732147+00:00 (in 9.999458 seconds)
notificationworker stdout | 2025-02-07 04:09:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:28 UTC)" (scheduled at 2025-02-07 04:09:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:09:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:09:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 18, 732974), True, datetime.datetime(2025, 2, 7, 4, 9, 18, 732974), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:09:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:09:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:09:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:09:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:09:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:03.419110+00:00 (in 39.997751 seconds)
repositorygcworker stdout | 2025-02-07 04:09:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:23 UTC)" (scheduled at 2025-02-07 04:09:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:09:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:09:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:09:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:09:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:40.424250+00:00 (in 16.000756 seconds)
gcworker stdout | 2025-02-07 04:09:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:54 UTC)" (scheduled at 2025-02-07 04:09:24.423055+00:00)
gcworker stdout | 2025-02-07 04:09:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:09:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:09:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:09:27,314 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:09:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:09:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:38.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 04:09:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:38 UTC)" (scheduled at 2025-02-07 04:09:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:09:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:09:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 28, 732871), True, datetime.datetime(2025, 2, 7, 4, 9, 28, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:09:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:09:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:09:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:09:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:09:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:44.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 04:09:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:44 UTC)" (scheduled at 2025-02-07 04:09:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:09:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:09:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:09:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:09:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:09:29,235 [235] [DEBUG] [app] Starting request: urn:request:947134a3-0194-4cec-8810-c5b8cd18170f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:29,235 [238] [DEBUG] [app] Starting request: urn:request:b12dbf2f-cd21-4b9d-8371-b7a6f0a39736 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:09:29,238 [244] [DEBUG] [app] Starting request: urn:request:cd39d6af-2ef8-4275-87e8-274eacddfa13 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:29,238 [239] [DEBUG] [app] Starting request: urn:request:9412f611-5b6f-4d4d-b3ab-d2f60ebc891b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:29,239 [239] [DEBUG] [app] Ending request: urn:request:9412f611-5b6f-4d4d-b3ab-d2f60ebc891b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9412f611-5b6f-4d4d-b3ab-d2f60ebc891b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:09:29,239 [244] [DEBUG] [app] Ending request: urn:request:cd39d6af-2ef8-4275-87e8-274eacddfa13 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cd39d6af-2ef8-4275-87e8-274eacddfa13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:09:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:09:29,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:09:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:29,243 [236] [DEBUG] [app] Starting request: urn:request:ab1ff552-b5c9-4a05-869d-e0d59f3dd661 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:29,243 [237] [DEBUG] [app] Starting request: urn:request:f8c8ca62-9c89-4aaf-bcc7-f6431333e4c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:29,243 [236] [DEBUG] [app] Ending request: urn:request:ab1ff552-b5c9-4a05-869d-e0d59f3dd661 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ab1ff552-b5c9-4a05-869d-e0d59f3dd661', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:09:29,243 [237] [DEBUG] [app] Ending request: urn:request:f8c8ca62-9c89-4aaf-bcc7-f6431333e4c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f8c8ca62-9c89-4aaf-bcc7-f6431333e4c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:09:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:09:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:09:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
repositoryactioncounter stdout | 2025-02-07 04:09:29,250 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
gunicorn-web stdout | 2025-02-07 04:09:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:29,255 [235] [DEBUG] [app] Ending request: urn:request:947134a3-0194-4cec-8810-c5b8cd18170f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:947134a3-0194-4cec-8810-c5b8cd18170f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:09:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:09:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:09:29,255 [238] [DEBUG] [app] Ending request: urn:request:b12dbf2f-cd21-4b9d-8371-b7a6f0a39736 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b12dbf2f-cd21-4b9d-8371-b7a6f0a39736', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:09:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:09:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
queuecleanupworker stdout | 2025-02-07 04:09:30,377 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:09:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:09:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:32.625945+00:00 (in 1.004310 seconds)
namespacegcworker stdout | 2025-02-07 04:09:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:31 UTC)" (scheduled at 2025-02-07 04:09:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:09:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:09:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 31, 621944), True, datetime.datetime(2025, 2, 7, 4, 9, 31, 621944), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:09:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:09:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:09:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:09:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:09:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:31.621131+00:00 (in 58.994653 seconds)
namespacegcworker stdout | 2025-02-07 04:09:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:32 UTC)" (scheduled at 2025-02-07 04:09:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:09:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:09:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:09:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:09:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:03.919286+00:00 (in 29.999519 seconds)
buildlogsarchiver stdout | 2025-02-07 04:09:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:03 UTC)" (scheduled at 2025-02-07 04:09:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:09:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 9, 33, 920029), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:09:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:09:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:09:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:09:34,640 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:09:35,690 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:56.713547+00:00 (in 18.997201 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:37 UTC)" (scheduled at 2025-02-07 04:09:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:09:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:09:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:48.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:09:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:48 UTC)" (scheduled at 2025-02-07 04:09:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:09:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:09:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 38, 732842), True, datetime.datetime(2025, 2, 7, 4, 9, 38, 732842), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:09:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:09:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:09:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:09:39,762 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:09:40,296 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:09:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:09:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:54.423055+00:00 (in 13.998377 seconds)
gcworker stdout | 2025-02-07 04:09:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:10 UTC)" (scheduled at 2025-02-07 04:09:40.424250+00:00)
gcworker stdout | 2025-02-07 04:09:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:09:40,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901080431, None, 1, 0])
gcworker stdout | 2025-02-07 04:09:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:09:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:09:40,838 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:09:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:09:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:11.831521+00:00 (in 29.999548 seconds)
autopruneworker stdout | 2025-02-07 04:09:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:11 UTC)" (scheduled at 2025-02-07 04:09:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:09:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897781836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:09:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:09:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:09:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:09:42,094 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:09:42,993 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:09:43,061 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:09:44,121 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:09:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:09:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:59.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:09:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:59 UTC)" (scheduled at 2025-02-07 04:09:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:09:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:09:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:09:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:09:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:09:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:09:44,235 [235] [DEBUG] [app] Starting request: urn:request:ff6f679f-34d3-4702-90a5-a9be5b0c5d17 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:44,235 [238] [DEBUG] [app] Starting request: urn:request:a46f5c86-7fc0-4a16-aeb5-d3d41eb2856d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:09:44,238 [239] [DEBUG] [app] Starting request: urn:request:3aea4b71-67d4-4523-951a-bc55280fc242 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:44,238 [246] [DEBUG] [app] Starting request: urn:request:95a57823-4a9b-4aee-9047-1a0288fcaf37 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:44,238 [239] [DEBUG] [app] Ending request: urn:request:3aea4b71-67d4-4523-951a-bc55280fc242 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3aea4b71-67d4-4523-951a-bc55280fc242', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:09:44,239 [246] [DEBUG] [app] Ending request: urn:request:95a57823-4a9b-4aee-9047-1a0288fcaf37 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:95a57823-4a9b-4aee-9047-1a0288fcaf37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:09:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:09:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:44,241 [235] [DEBUG] [app] Starting request: urn:request:727f398a-4e19-4c23-a567-7b656e3b6bfa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:44,242 [235] [DEBUG] [app] Ending request: urn:request:727f398a-4e19-4c23-a567-7b656e3b6bfa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:727f398a-4e19-4c23-a567-7b656e3b6bfa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:09:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:44,243 [237] [DEBUG] [app] Starting request: urn:request:90bc27ae-c8e0-4785-ab2c-e3f98412835e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:44,243 [237] [DEBUG] [app] Ending request: urn:request:90bc27ae-c8e0-4785-ab2c-e3f98412835e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:90bc27ae-c8e0-4785-ab2c-e3f98412835e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:09:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:09:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:09:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:09:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:44,254 [235] [DEBUG] [app] Ending request: urn:request:ff6f679f-34d3-4702-90a5-a9be5b0c5d17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ff6f679f-34d3-4702-90a5-a9be5b0c5d17', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:09:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:09:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:09:44,255 [238] [DEBUG] [app] Ending request: urn:request:a46f5c86-7fc0-4a16-aeb5-d3d41eb2856d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a46f5c86-7fc0-4a16-aeb5-d3d41eb2856d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:09:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 04:09:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:44,506 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:09:45,129 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:09:45,207 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:09:45,300 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:09:45,699 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:09:46,079 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:09:46,221 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:09:46,267 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:09:46,437 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:09:46,442 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:09:46,444 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:09:46,447 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:09:46,450 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:09:47,458 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:09:47,461 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:09:47,465 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:09:47,892 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:09:48,091 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:09:48,337 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:09:48,559 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:09:48,565 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:09:48,569 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:09:48,572 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:09:48,575 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:09:48,578 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:09:48,581 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:09:48,585 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:09:48,587 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:09:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:09:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:50.814677+00:00 (in 2.082085 seconds)
notificationworker stdout | 2025-02-07 04:09:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:58 UTC)" (scheduled at 2025-02-07 04:09:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:09:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:09:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 48, 732908), True, datetime.datetime(2025, 2, 7, 4, 9, 48, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:09:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:09:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:09:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:09:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:09:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:09:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:58.732147+00:00 (in 7.917015 seconds)
notificationworker stdout | 2025-02-07 04:09:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:50 UTC)" (scheduled at 2025-02-07 04:09:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:09:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:09:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:09:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:09:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:09:59.937899+00:00 (in 6.999821 seconds)
exportactionlogsworker stdout | 2025-02-07 04:09:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:14:52 UTC)" (scheduled at 2025-02-07 04:09:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:09:52,939 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 9, 52, 938361), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:09:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 9, 52, 938361), True, datetime.datetime(2025, 2, 7, 4, 9, 52, 938361), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:09:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 9, 52, 938361), True, datetime.datetime(2025, 2, 7, 4, 9, 52, 938361), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 9, 52, 938361), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:09:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:09:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:14:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:09:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:09:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:10.424250+00:00 (in 16.000708 seconds)
gcworker stdout | 2025-02-07 04:09:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:24 UTC)" (scheduled at 2025-02-07 04:09:54.423055+00:00)
gcworker stdout | 2025-02-07 04:09:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:09:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:09:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:09:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:56.118855+00:00 (in 59.999509 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:09:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:56 UTC)" (scheduled at 2025-02-07 04:09:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:09:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:09:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:09:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:37.715878+00:00 (in 41.001885 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:56 UTC)" (scheduled at 2025-02-07 04:09:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:09:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:09:57,350 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:09:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:09:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:08.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:09:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:08 UTC)" (scheduled at 2025-02-07 04:09:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:09:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:09:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 9, 58, 732878), True, datetime.datetime(2025, 2, 7, 4, 9, 58, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:09:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:09:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:09:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:09:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:09:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:14.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 04:09:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:14 UTC)" (scheduled at 2025-02-07 04:09:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:09:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:09:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:09:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:09:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:09:59,236 [236] [DEBUG] [app] Starting request: urn:request:cd1b72cd-9d38-4ef3-a2b3-26210e4b40fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:59,236 [237] [DEBUG] [app] Starting request: urn:request:fc0e07a9-3b1b-4307-ade9-f869aebf4762 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:09:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:09:59,239 [245] [DEBUG] [app] Starting request: urn:request:d147b634-75c2-4d6b-8318-dbd0073d6aa0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:59,239 [239] [DEBUG] [app] Starting request: urn:request:1818900c-aeee-44a8-b4b2-8cba6831d8c5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:09:59,240 [245] [DEBUG] [app] Ending request: urn:request:d147b634-75c2-4d6b-8318-dbd0073d6aa0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d147b634-75c2-4d6b-8318-dbd0073d6aa0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:09:59,240 [239] [DEBUG] [app] Ending request: urn:request:1818900c-aeee-44a8-b4b2-8cba6831d8c5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1818900c-aeee-44a8-b4b2-8cba6831d8c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:09:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:09:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:09:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:09:59,243 [237] [DEBUG] [app] Starting request: urn:request:dbff41c7-2bd8-421e-beed-09007d7136e0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:59,243 [235] [DEBUG] [app] Starting request: urn:request:eeadddcf-d846-4453-9de4-bdb4cdfe903a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:09:59,243 [237] [DEBUG] [app] Ending request: urn:request:dbff41c7-2bd8-421e-beed-09007d7136e0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dbff41c7-2bd8-421e-beed-09007d7136e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:09:59,243 [235] [DEBUG] [app] Ending request: urn:request:eeadddcf-d846-4453-9de4-bdb4cdfe903a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eeadddcf-d846-4453-9de4-bdb4cdfe903a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:09:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:09:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:09:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:09:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:09:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:09:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:09:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:09:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:09:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:09:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:09:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:09:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:09:59,255 [237] [DEBUG] [app] Ending request: urn:request:fc0e07a9-3b1b-4307-ade9-f869aebf4762 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fc0e07a9-3b1b-4307-ade9-f869aebf4762', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:09:59,255 [236] [DEBUG] [app] Ending request: urn:request:cd1b72cd-9d38-4ef3-a2b3-26210e4b40fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd1b72cd-9d38-4ef3-a2b3-26210e4b40fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:09:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:09:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:09:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:09:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:09:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:09:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:09:59,287 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:09:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:09:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:04.936688+00:00 (in 4.998299 seconds)
exportactionlogsworker stdout | 2025-02-07 04:09:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:59 UTC)" (scheduled at 2025-02-07 04:09:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:09:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:09:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:10:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:10:00,413 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:10:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:10:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:23.420886+00:00 (in 20.001308 seconds)
repositorygcworker stdout | 2025-02-07 04:10:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:03 UTC)" (scheduled at 2025-02-07 04:10:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:10:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:10:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 3, 419789), True, datetime.datetime(2025, 2, 7, 4, 10, 3, 419789), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:10:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:10:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:10:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:10:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:10:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:33.919286+00:00 (in 29.999526 seconds)
buildlogsarchiver stdout | 2025-02-07 04:10:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:33 UTC)" (scheduled at 2025-02-07 04:10:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:10:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 10, 3, 920018), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:10:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:10:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:10:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:10:04,677 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:10:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:10:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:59.937899+00:00 (in 55.000783 seconds)
exportactionlogsworker stdout | 2025-02-07 04:10:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:04 UTC)" (scheduled at 2025-02-07 04:10:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:10:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:10:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 4, 937442), True, datetime.datetime(2025, 2, 7, 4, 10, 4, 937442), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:10:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:10:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:10:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:10:05,727 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:10:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:10:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:18.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 04:10:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:18 UTC)" (scheduled at 2025-02-07 04:10:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:10:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:10:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 8, 732893), True, datetime.datetime(2025, 2, 7, 4, 10, 8, 732893), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:10:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:10:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:10:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:10:09,773 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:10:10,331 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:10:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:10:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:24.423055+00:00 (in 13.998339 seconds)
gcworker stdout | 2025-02-07 04:10:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:40 UTC)" (scheduled at 2025-02-07 04:10:10.424250+00:00)
gcworker stdout | 2025-02-07 04:10:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:10:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901110431, None, 1, 0])
gcworker stdout | 2025-02-07 04:10:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:10:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:10:10,874 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:10:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:10:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:41.831521+00:00 (in 29.999544 seconds)
autopruneworker stdout | 2025-02-07 04:10:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:41 UTC)" (scheduled at 2025-02-07 04:10:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:10:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897811836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:10:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:10:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:10:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:10:12,118 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:10:13,013 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:10:13,078 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:10:14,129 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:10:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:10:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:29.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 04:10:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:29 UTC)" (scheduled at 2025-02-07 04:10:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:10:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:10:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:10:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:10:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:10:14,236 [235] [DEBUG] [app] Starting request: urn:request:3c9e6ca7-3c94-467e-815a-0af15455c22e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:14,236 [238] [DEBUG] [app] Starting request: urn:request:b3233ab7-6f76-4d2b-bf51-29fa3683b76b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:10:14,239 [245] [DEBUG] [app] Starting request: urn:request:f0518ad2-d918-4c9d-bede-3e0134cbdd12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:14,239 [246] [DEBUG] [app] Starting request: urn:request:5f226e02-dd6b-4bf8-b400-6628c38bec17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:14,240 [245] [DEBUG] [app] Ending request: urn:request:f0518ad2-d918-4c9d-bede-3e0134cbdd12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0518ad2-d918-4c9d-bede-3e0134cbdd12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:10:14,240 [246] [DEBUG] [app] Ending request: urn:request:5f226e02-dd6b-4bf8-b400-6628c38bec17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f226e02-dd6b-4bf8-b400-6628c38bec17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:10:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:10:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:14,243 [237] [DEBUG] [app] Starting request: urn:request:d5df7576-2951-4a2a-b4e8-1439c3ed633d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:14,243 [236] [DEBUG] [app] Starting request: urn:request:3bc7d1a5-cb96-4c74-a80c-168b113568fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:14,244 [236] [DEBUG] [app] Ending request: urn:request:3bc7d1a5-cb96-4c74-a80c-168b113568fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3bc7d1a5-cb96-4c74-a80c-168b113568fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:10:14,244 [237] [DEBUG] [app] Ending request: urn:request:d5df7576-2951-4a2a-b4e8-1439c3ed633d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5df7576-2951-4a2a-b4e8-1439c3ed633d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:10:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:14,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:14,256 [235] [DEBUG] [app] Ending request: urn:request:3c9e6ca7-3c94-467e-815a-0af15455c22e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3c9e6ca7-3c94-467e-815a-0af15455c22e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:14,256 [238] [DEBUG] [app] Ending request: urn:request:b3233ab7-6f76-4d2b-bf51-29fa3683b76b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b3233ab7-6f76-4d2b-bf51-29fa3683b76b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:10:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:10:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:10:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:14,522 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:10:15,140 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:10:15,235 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:10:15,322 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:10:15,719 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:10:16,116 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:10:16,257 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:10:16,281 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:10:16,445 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:10:16,448 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:10:16,451 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:10:16,455 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:10:16,458 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:10:17,467 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:10:17,470 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:10:17,473 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:10:17,920 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:10:18,106 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:10:18,353 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:10:18,573 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:10:18,576 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:10:18,580 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:10:18,584 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:10:18,588 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:10:18,591 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:10:18,594 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:10:18,597 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:10:18,599 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:10:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:10:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:28.732147+00:00 (in 9.999522 seconds)
notificationworker stdout | 2025-02-07 04:10:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:28 UTC)" (scheduled at 2025-02-07 04:10:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:10:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:10:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 18, 732938), True, datetime.datetime(2025, 2, 7, 4, 10, 18, 732938), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:10:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:10:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:10:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:10:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:10:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:03.419110+00:00 (in 39.997728 seconds)
repositorygcworker stdout | 2025-02-07 04:10:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:23 UTC)" (scheduled at 2025-02-07 04:10:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:10:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:10:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:10:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:10:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:40.424250+00:00 (in 16.000720 seconds)
gcworker stdout | 2025-02-07 04:10:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:54 UTC)" (scheduled at 2025-02-07 04:10:24.423055+00:00)
gcworker stdout | 2025-02-07 04:10:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:10:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:10:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:10:27,386 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:10:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:10:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:38.732147+00:00 (in 9.999476 seconds)
notificationworker stdout | 2025-02-07 04:10:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:38 UTC)" (scheduled at 2025-02-07 04:10:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:10:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:10:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 28, 732979), True, datetime.datetime(2025, 2, 7, 4, 10, 28, 732979), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:10:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:10:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:10:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:10:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:10:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:44.225378+00:00 (in 14.999538 seconds)
quotatotalworker stdout | 2025-02-07 04:10:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:44 UTC)" (scheduled at 2025-02-07 04:10:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:10:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:10:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:10:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:10:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:10:29,236 [238] [DEBUG] [app] Starting request: urn:request:a6f15c36-8207-40d1-b134-b5af5f9a26ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:29,236 [237] [DEBUG] [app] Starting request: urn:request:0773fa32-4479-4894-891d-e8a592a89933 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:10:29,238 [239] [DEBUG] [app] Starting request: urn:request:07cbfd1b-9117-4c41-b9f6-bc8d259e3cbe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:29,238 [245] [DEBUG] [app] Starting request: urn:request:11b7542c-5c60-4b14-a662-5e1b1b5b49d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:29,239 [239] [DEBUG] [app] Ending request: urn:request:07cbfd1b-9117-4c41-b9f6-bc8d259e3cbe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07cbfd1b-9117-4c41-b9f6-bc8d259e3cbe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:10:29,239 [245] [DEBUG] [app] Ending request: urn:request:11b7542c-5c60-4b14-a662-5e1b1b5b49d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11b7542c-5c60-4b14-a662-5e1b1b5b49d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:10:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:10:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:29,242 [237] [DEBUG] [app] Starting request: urn:request:79a62074-663c-43c7-b8fa-0cbe0b2fca6f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:29,242 [237] [DEBUG] [app] Ending request: urn:request:79a62074-663c-43c7-b8fa-0cbe0b2fca6f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79a62074-663c-43c7-b8fa-0cbe0b2fca6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:10:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:10:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:29,243 [236] [DEBUG] [app] Starting request: urn:request:ba6d849e-264a-4039-8303-46666c594200 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:29,244 [236] [DEBUG] [app] Ending request: urn:request:ba6d849e-264a-4039-8303-46666c594200 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba6d849e-264a-4039-8303-46666c594200', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:10:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:29,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:29,254 [238] [DEBUG] [app] Ending request: urn:request:a6f15c36-8207-40d1-b134-b5af5f9a26ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6f15c36-8207-40d1-b134-b5af5f9a26ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:10:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:10:29,255 [237] [DEBUG] [app] Ending request: urn:request:0773fa32-4479-4894-891d-e8a592a89933 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0773fa32-4479-4894-891d-e8a592a89933', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:10:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:10:29,323 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:10:30,434 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:10:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:10:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:32.625945+00:00 (in 1.004307 seconds)
namespacegcworker stdout | 2025-02-07 04:10:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:31 UTC)" (scheduled at 2025-02-07 04:10:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:10:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:10:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 31, 621906), True, datetime.datetime(2025, 2, 7, 4, 10, 31, 621906), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:10:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:10:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:10:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:10:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:10:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:31.621131+00:00 (in 58.994720 seconds)
namespacegcworker stdout | 2025-02-07 04:10:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:32 UTC)" (scheduled at 2025-02-07 04:10:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:10:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:10:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:10:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:10:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:03.919286+00:00 (in 29.999526 seconds)
buildlogsarchiver stdout | 2025-02-07 04:10:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:03 UTC)" (scheduled at 2025-02-07 04:10:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:10:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 10, 33, 920021), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:10:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:10:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:10:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:10:34,712 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:10:35,762 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:56.713547+00:00 (in 18.997196 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:37 UTC)" (scheduled at 2025-02-07 04:10:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:10:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:10:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:48.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 04:10:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:48 UTC)" (scheduled at 2025-02-07 04:10:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:10:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:10:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 38, 732923), True, datetime.datetime(2025, 2, 7, 4, 10, 38, 732923), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:10:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:10:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:10:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:10:39,809 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:10:40,367 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:10:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:10:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:54.423055+00:00 (in 13.998330 seconds)
gcworker stdout | 2025-02-07 04:10:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:10 UTC)" (scheduled at 2025-02-07 04:10:40.424250+00:00)
gcworker stdout | 2025-02-07 04:10:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:10:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901140431, None, 1, 0])
gcworker stdout | 2025-02-07 04:10:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:10:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:10:40,890 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:10:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:10:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:11.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 04:10:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:11 UTC)" (scheduled at 2025-02-07 04:10:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:10:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897841836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:10:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:10:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:10:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:10:42,129 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:10:43,034 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:10:43,115 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:10:44,144 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:10:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:10:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:59.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 04:10:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:59 UTC)" (scheduled at 2025-02-07 04:10:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:10:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:10:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:10:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:10:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:10:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:10:44,235 [237] [DEBUG] [app] Starting request: urn:request:a2faf177-6a9d-4bca-a487-98b533645822 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:44,235 [238] [DEBUG] [app] Starting request: urn:request:bbb070cd-6e47-43e3-a886-5f8d7ce078ed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:10:44,238 [246] [DEBUG] [app] Starting request: urn:request:11ed698a-265a-4036-bedf-b887de9ddf69 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:44,238 [245] [DEBUG] [app] Starting request: urn:request:5dc258f2-a030-41ce-bb11-0d4f8fa6647d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:44,238 [245] [DEBUG] [app] Ending request: urn:request:5dc258f2-a030-41ce-bb11-0d4f8fa6647d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5dc258f2-a030-41ce-bb11-0d4f8fa6647d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:10:44,238 [246] [DEBUG] [app] Ending request: urn:request:11ed698a-265a-4036-bedf-b887de9ddf69 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11ed698a-265a-4036-bedf-b887de9ddf69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:10:44,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:10:44,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:44,238 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:44,241 [238] [DEBUG] [app] Starting request: urn:request:db148f87-5dee-447b-9322-8d9905cd43fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:44,241 [235] [DEBUG] [app] Starting request: urn:request:8399f46c-6d32-41cd-bfe3-c0f0bf381553 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:44,241 [238] [DEBUG] [app] Ending request: urn:request:db148f87-5dee-447b-9322-8d9905cd43fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db148f87-5dee-447b-9322-8d9905cd43fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:10:44,241 [235] [DEBUG] [app] Ending request: urn:request:8399f46c-6d32-41cd-bfe3-c0f0bf381553 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8399f46c-6d32-41cd-bfe3-c0f0bf381553', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:10:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:10:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:10:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:44,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:44,253 [237] [DEBUG] [app] Ending request: urn:request:a2faf177-6a9d-4bca-a487-98b533645822 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2faf177-6a9d-4bca-a487-98b533645822', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:44,253 [238] [DEBUG] [app] Ending request: urn:request:bbb070cd-6e47-43e3-a886-5f8d7ce078ed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bbb070cd-6e47-43e3-a886-5f8d7ce078ed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:10:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:10:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:10:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:44,558 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:10:45,154 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:10:45,271 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:10:45,359 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:10:45,755 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:10:46,152 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:10:46,274 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:10:46,317 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:10:46,454 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:10:46,457 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:10:46,460 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:10:46,463 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:10:46,465 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:10:47,475 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:10:47,478 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:10:47,480 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:10:47,931 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:10:48,128 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:10:48,386 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:10:48,585 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:10:48,587 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:10:48,591 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:10:48,594 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:10:48,596 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:10:48,600 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:10:48,605 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:10:48,608 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:10:48,611 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:10:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:10:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:50.814677+00:00 (in 2.082057 seconds)
notificationworker stdout | 2025-02-07 04:10:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:58 UTC)" (scheduled at 2025-02-07 04:10:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:10:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:10:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 48, 732923), True, datetime.datetime(2025, 2, 7, 4, 10, 48, 732923), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:10:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:10:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:10:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:10:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:10:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:10:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:10:58.732147+00:00 (in 7.916986 seconds)
notificationworker stdout | 2025-02-07 04:10:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:50 UTC)" (scheduled at 2025-02-07 04:10:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:10:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:10:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:10:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:10:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:10.424250+00:00 (in 16.000717 seconds)
gcworker stdout | 2025-02-07 04:10:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:24 UTC)" (scheduled at 2025-02-07 04:10:54.423055+00:00)
gcworker stdout | 2025-02-07 04:10:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:10:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:10:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:10:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:56.118855+00:00 (in 59.999504 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:10:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:56 UTC)" (scheduled at 2025-02-07 04:10:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:10:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:10:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:10:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:37.715878+00:00 (in 41.001877 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:56 UTC)" (scheduled at 2025-02-07 04:10:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:10:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:10:57,422 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:10:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:10:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:08.732147+00:00 (in 9.999486 seconds)
notificationworker stdout | 2025-02-07 04:10:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:08 UTC)" (scheduled at 2025-02-07 04:10:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:10:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:10:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 10, 58, 732963), True, datetime.datetime(2025, 2, 7, 4, 10, 58, 732963), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:10:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:10:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:10:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:10:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:10:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:14.225378+00:00 (in 14.999602 seconds)
quotatotalworker stdout | 2025-02-07 04:10:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:14 UTC)" (scheduled at 2025-02-07 04:10:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:10:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:10:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:10:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:10:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:10:59,235 [238] [DEBUG] [app] Starting request: urn:request:0ea45815-3049-498b-a89f-1783328164a2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:59,235 [237] [DEBUG] [app] Starting request: urn:request:c9b0188e-55d4-4ad7-a7d2-d0ee0b1fc4c5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:10:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:10:59,238 [246] [DEBUG] [app] Starting request: urn:request:02a0f02e-9143-4aa5-85e7-5323aaf98e2d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:59,238 [246] [DEBUG] [app] Ending request: urn:request:02a0f02e-9143-4aa5-85e7-5323aaf98e2d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02a0f02e-9143-4aa5-85e7-5323aaf98e2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:10:59,239 [239] [DEBUG] [app] Starting request: urn:request:e2a2faf0-8193-46de-a36a-a893fd732fbe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:10:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:10:59,239 [239] [DEBUG] [app] Ending request: urn:request:e2a2faf0-8193-46de-a36a-a893fd732fbe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e2a2faf0-8193-46de-a36a-a893fd732fbe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:10:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:10:59,242 [235] [DEBUG] [app] Starting request: urn:request:18bf452f-ac73-4b1b-a3ee-2f93c95c72d6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:59,242 [235] [DEBUG] [app] Ending request: urn:request:18bf452f-ac73-4b1b-a3ee-2f93c95c72d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18bf452f-ac73-4b1b-a3ee-2f93c95c72d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:10:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:10:59,243 [237] [DEBUG] [app] Starting request: urn:request:dfd37eca-5ba1-47ea-bb4f-50951167c0dc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:10:59,243 [237] [DEBUG] [app] Ending request: urn:request:dfd37eca-5ba1-47ea-bb4f-50951167c0dc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dfd37eca-5ba1-47ea-bb4f-50951167c0dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:10:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:10:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:10:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:10:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:10:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:10:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:10:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:10:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:10:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:10:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:10:59,255 [237] [DEBUG] [app] Ending request: urn:request:c9b0188e-55d4-4ad7-a7d2-d0ee0b1fc4c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9b0188e-55d4-4ad7-a7d2-d0ee0b1fc4c5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:59,255 [238] [DEBUG] [app] Ending request: urn:request:0ea45815-3049-498b-a89f-1783328164a2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ea45815-3049-498b-a89f-1783328164a2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:10:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:10:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:10:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:10:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:10:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:10:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:10:59,359 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:10:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:10:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:04.936688+00:00 (in 4.998363 seconds)
exportactionlogsworker stdout | 2025-02-07 04:10:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:59 UTC)" (scheduled at 2025-02-07 04:10:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:10:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:10:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:11:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:11:00,470 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:11:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:11:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:23.420886+00:00 (in 20.001285 seconds)
repositorygcworker stdout | 2025-02-07 04:11:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:03 UTC)" (scheduled at 2025-02-07 04:11:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:11:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:11:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 3, 419809), True, datetime.datetime(2025, 2, 7, 4, 11, 3, 419809), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:11:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:11:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:11:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:11:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:11:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:33.919286+00:00 (in 29.999542 seconds)
buildlogsarchiver stdout | 2025-02-07 04:11:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:33 UTC)" (scheduled at 2025-02-07 04:11:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:11:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 11, 3, 919997), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:11:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:11:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:11:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:11:04,748 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:11:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:11:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:59.937899+00:00 (in 55.000798 seconds)
exportactionlogsworker stdout | 2025-02-07 04:11:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:04 UTC)" (scheduled at 2025-02-07 04:11:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:11:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:11:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 4, 937390), True, datetime.datetime(2025, 2, 7, 4, 11, 4, 937390), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:11:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:11:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:11:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:11:05,788 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:11:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:11:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:18.732147+00:00 (in 9.999466 seconds)
notificationworker stdout | 2025-02-07 04:11:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:18 UTC)" (scheduled at 2025-02-07 04:11:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:11:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:11:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 8, 733004), True, datetime.datetime(2025, 2, 7, 4, 11, 8, 733004), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:11:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:11:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:11:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:11:09,818 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:11:10,404 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:11:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:11:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:24.423055+00:00 (in 13.998387 seconds)
gcworker stdout | 2025-02-07 04:11:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:40 UTC)" (scheduled at 2025-02-07 04:11:10.424250+00:00)
gcworker stdout | 2025-02-07 04:11:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:11:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901170431, None, 1, 0])
gcworker stdout | 2025-02-07 04:11:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:11:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:40 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:11:10,899 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:11:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:11:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:41.831521+00:00 (in 29.999557 seconds)
autopruneworker stdout | 2025-02-07 04:11:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:41 UTC)" (scheduled at 2025-02-07 04:11:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:11:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897871836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:11:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:11:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:11:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:11:12,141 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:11:13,071 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:11:13,123 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:11:14,182 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:11:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:11:14,226 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:29.225378+00:00 (in 14.999374 seconds)
quotatotalworker stdout | 2025-02-07 04:11:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:29 UTC)" (scheduled at 2025-02-07 04:11:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:11:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:11:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:11:14,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:11:14,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:11:14,236 [238] [DEBUG] [app] Starting request: urn:request:837a4e68-7d71-4e2d-a326-71429662239d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:14,236 [237] [DEBUG] [app] Starting request: urn:request:ab0a07ea-ac15-4cd8-a634-8c8991ed55cc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:11:14,240 [239] [DEBUG] [app] Starting request: urn:request:c0172ffa-1b1a-493e-a0ee-245ad237dcb5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:14,240 [245] [DEBUG] [app] Starting request: urn:request:f86a7a97-b421-4847-9695-d2f540f43f05 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:14,240 [245] [DEBUG] [app] Ending request: urn:request:f86a7a97-b421-4847-9695-d2f540f43f05 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f86a7a97-b421-4847-9695-d2f540f43f05', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:11:14,240 [239] [DEBUG] [app] Ending request: urn:request:c0172ffa-1b1a-493e-a0ee-245ad237dcb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c0172ffa-1b1a-493e-a0ee-245ad237dcb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:11:14,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:11:14,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:11:14,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:11:14,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:14,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:14,244 [237] [DEBUG] [app] Starting request: urn:request:ad5304d3-2b8d-4007-8b54-0c55623802ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:14,244 [237] [DEBUG] [app] Ending request: urn:request:ad5304d3-2b8d-4007-8b54-0c55623802ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad5304d3-2b8d-4007-8b54-0c55623802ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:11:14,244 [236] [DEBUG] [app] Starting request: urn:request:b82080c7-dd6a-492b-8113-bb98ccf872d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:11:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:14,244 [236] [DEBUG] [app] Ending request: urn:request:b82080c7-dd6a-492b-8113-bb98ccf872d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b82080c7-dd6a-492b-8113-bb98ccf872d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:11:14,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:14,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:11:14,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:11:14,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:11:14,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:14,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:14,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:14,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:14,256 [237] [DEBUG] [app] Ending request: urn:request:ab0a07ea-ac15-4cd8-a634-8c8991ed55cc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab0a07ea-ac15-4cd8-a634-8c8991ed55cc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:14,256 [238] [DEBUG] [app] Ending request: urn:request:837a4e68-7d71-4e2d-a326-71429662239d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:837a4e68-7d71-4e2d-a326-71429662239d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:11:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:14,579 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:11:15,174 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:11:15,308 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:11:15,395 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:11:15,772 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:11:16,174 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:11:16,296 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:11:16,327 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:11:16,461 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:11:16,465 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:11:16,468 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:11:16,470 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:11:16,472 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:11:17,484 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:11:17,487 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:11:17,490 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:11:17,967 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:11:18,146 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:11:18,422 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:11:18,596 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:11:18,598 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:11:18,602 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:11:18,604 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:11:18,607 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:11:18,610 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:11:18,613 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:11:18,616 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:11:18,618 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:11:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:11:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:28.732147+00:00 (in 9.999512 seconds)
notificationworker stdout | 2025-02-07 04:11:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:28 UTC)" (scheduled at 2025-02-07 04:11:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:11:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:11:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 18, 732908), True, datetime.datetime(2025, 2, 7, 4, 11, 18, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:11:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:11:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:11:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:11:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:11:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:03.419110+00:00 (in 39.997751 seconds)
repositorygcworker stdout | 2025-02-07 04:11:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:23 UTC)" (scheduled at 2025-02-07 04:11:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:11:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:11:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:11:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:11:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:40.424250+00:00 (in 16.000735 seconds)
gcworker stdout | 2025-02-07 04:11:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:54 UTC)" (scheduled at 2025-02-07 04:11:24.423055+00:00)
gcworker stdout | 2025-02-07 04:11:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:11:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:11:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:11:27,436 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:11:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:11:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:38.732147+00:00 (in 9.999529 seconds)
notificationworker stdout | 2025-02-07 04:11:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:38 UTC)" (scheduled at 2025-02-07 04:11:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:11:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:11:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 28, 732904), True, datetime.datetime(2025, 2, 7, 4, 11, 28, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:11:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:11:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:11:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:11:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:11:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:44.225378+00:00 (in 14.999502 seconds)
quotatotalworker stdout | 2025-02-07 04:11:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:44 UTC)" (scheduled at 2025-02-07 04:11:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:11:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:11:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:11:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:11:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:11:29,235 [237] [DEBUG] [app] Starting request: urn:request:d23b4c10-e166-40bf-9bce-5a149808b8b8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:29,235 [235] [DEBUG] [app] Starting request: urn:request:66ff9dbd-44b7-48ec-a805-d4425b9845ab (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:11:29,238 [239] [DEBUG] [app] Starting request: urn:request:391213f2-4c6b-49f6-93f1-16c9dfe7280f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:29,238 [246] [DEBUG] [app] Starting request: urn:request:37d63a58-02a3-4fe6-9129-cf5910ad7f72 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:29,238 [239] [DEBUG] [app] Ending request: urn:request:391213f2-4c6b-49f6-93f1-16c9dfe7280f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:391213f2-4c6b-49f6-93f1-16c9dfe7280f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:11:29,238 [246] [DEBUG] [app] Ending request: urn:request:37d63a58-02a3-4fe6-9129-cf5910ad7f72 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:37d63a58-02a3-4fe6-9129-cf5910ad7f72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:11:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:11:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:11:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:29,241 [235] [DEBUG] [app] Starting request: urn:request:20739dd4-8d04-45e4-935f-d46a483fab45 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:29,241 [235] [DEBUG] [app] Ending request: urn:request:20739dd4-8d04-45e4-935f-d46a483fab45 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:20739dd4-8d04-45e4-935f-d46a483fab45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:11:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:11:29,242 [238] [DEBUG] [app] Starting request: urn:request:3f6e2961-71a4-41b9-9f30-6b325df12dfe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:29,242 [238] [DEBUG] [app] Ending request: urn:request:3f6e2961-71a4-41b9-9f30-6b325df12dfe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f6e2961-71a4-41b9-9f30-6b325df12dfe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:11:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:11:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:11:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:11:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:29,253 [235] [DEBUG] [app] Ending request: urn:request:66ff9dbd-44b7-48ec-a805-d4425b9845ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66ff9dbd-44b7-48ec-a805-d4425b9845ab', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 04:11:29,254 [237] [DEBUG] [app] Ending request: urn:request:d23b4c10-e166-40bf-9bce-5a149808b8b8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d23b4c10-e166-40bf-9bce-5a149808b8b8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:11:29,395 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:11:30,506 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:11:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:11:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:32.625945+00:00 (in 1.004357 seconds)
namespacegcworker stdout | 2025-02-07 04:11:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:31 UTC)" (scheduled at 2025-02-07 04:11:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:11:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:11:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 31, 621859), True, datetime.datetime(2025, 2, 7, 4, 11, 31, 621859), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:11:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:11:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:11:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:11:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:11:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:13.624301+00:00 (in 40.997905 seconds)
namespacegcworker stdout | 2025-02-07 04:11:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:32 UTC)" (scheduled at 2025-02-07 04:11:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:11:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:11:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:11:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:11:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:03.919286+00:00 (in 29.999520 seconds)
buildlogsarchiver stdout | 2025-02-07 04:11:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:03 UTC)" (scheduled at 2025-02-07 04:11:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:11:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 11, 33, 920016), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:11:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:11:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:11:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:11:34,771 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:11:35,822 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:56.713547+00:00 (in 18.997209 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:37 UTC)" (scheduled at 2025-02-07 04:11:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:11:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:11:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:48.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 04:11:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:48 UTC)" (scheduled at 2025-02-07 04:11:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:11:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:11:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 38, 732865), True, datetime.datetime(2025, 2, 7, 4, 11, 38, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:11:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:11:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:11:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:11:39,825 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:11:40,416 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
gcworker stdout | 2025-02-07 04:11:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:11:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:54.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 04:11:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:10 UTC)" (scheduled at 2025-02-07 04:11:40.424250+00:00)
gcworker stdout | 2025-02-07 04:11:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:11:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901200431, None, 1, 0])
gcworker stdout | 2025-02-07 04:11:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:11:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:10 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:11:40,932 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:11:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:11:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:11.831521+00:00 (in 29.999538 seconds)
autopruneworker stdout | 2025-02-07 04:11:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:11 UTC)" (scheduled at 2025-02-07 04:11:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:11:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897901836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:11:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:11:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:11:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:11:42,177 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:11:43,089 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:11:43,140 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:11:44,197 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:11:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:11:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:59.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 04:11:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:59 UTC)" (scheduled at 2025-02-07 04:11:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:11:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:11:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:11:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:11:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:11:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:11:44,235 [235] [DEBUG] [app] Starting request: urn:request:bf475fc3-4919-47c3-a764-c94104709284 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:44,235 [236] [DEBUG] [app] Starting request: urn:request:3bd35a85-1bcf-4102-b77e-d86ec5112845 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:11:44,238 [245] [DEBUG] [app] Starting request: urn:request:fbfdd464-67e1-405c-bc57-a2837e9c5e54 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:44,238 [246] [DEBUG] [app] Starting request: urn:request:1412d135-f932-40d7-a0a9-b5bab7c61b7f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:44,239 [245] [DEBUG] [app] Ending request: urn:request:fbfdd464-67e1-405c-bc57-a2837e9c5e54 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fbfdd464-67e1-405c-bc57-a2837e9c5e54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:11:44,239 [246] [DEBUG] [app] Ending request: urn:request:1412d135-f932-40d7-a0a9-b5bab7c61b7f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1412d135-f932-40d7-a0a9-b5bab7c61b7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:11:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:11:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:11:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:44,241 [237] [DEBUG] [app] Starting request: urn:request:04507086-4f5d-4f68-a142-bed6c501e6bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:44,242 [237] [DEBUG] [app] Ending request: urn:request:04507086-4f5d-4f68-a142-bed6c501e6bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:04507086-4f5d-4f68-a142-bed6c501e6bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:11:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:11:44,243 [235] [DEBUG] [app] Starting request: urn:request:698a5746-64b4-4ccf-a8c4-4b9274e1027f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:44,243 [235] [DEBUG] [app] Ending request: urn:request:698a5746-64b4-4ccf-a8c4-4b9274e1027f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:698a5746-64b4-4ccf-a8c4-4b9274e1027f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:11:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:11:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:11:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:11:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:44,255 [236] [DEBUG] [app] Ending request: urn:request:3bd35a85-1bcf-4102-b77e-d86ec5112845 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3bd35a85-1bcf-4102-b77e-d86ec5112845', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:44,255 [235] [DEBUG] [app] Ending request: urn:request:bf475fc3-4919-47c3-a764-c94104709284 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf475fc3-4919-47c3-a764-c94104709284', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:11:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:44,609 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:11:45,210 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:11:45,317 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:11:45,422 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:11:45,782 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:11:46,194 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:11:46,331 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:11:46,352 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:11:46,470 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:11:46,474 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:11:46,477 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:11:46,479 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:11:46,482 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:11:47,493 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:11:47,497 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:11:47,499 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:11:47,995 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:11:48,168 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:11:48,434 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:11:48,606 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:11:48,611 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:11:48,614 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:11:48,616 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:11:48,619 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:11:48,621 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:11:48,624 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:11:48,628 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:11:48,630 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:11:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:11:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:50.814677+00:00 (in 2.082087 seconds)
notificationworker stdout | 2025-02-07 04:11:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:58 UTC)" (scheduled at 2025-02-07 04:11:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:11:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:11:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 48, 732866), True, datetime.datetime(2025, 2, 7, 4, 11, 48, 732866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:11:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:11:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:11:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:11:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:11:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:11:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:11:58.732147+00:00 (in 7.917015 seconds)
notificationworker stdout | 2025-02-07 04:11:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:50 UTC)" (scheduled at 2025-02-07 04:11:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:11:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:11:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:11:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:11:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:10.424250+00:00 (in 16.000685 seconds)
gcworker stdout | 2025-02-07 04:11:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:24 UTC)" (scheduled at 2025-02-07 04:11:54.423055+00:00)
gcworker stdout | 2025-02-07 04:11:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:11:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:11:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:11:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:56.118855+00:00 (in 59.999463 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:11:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:56 UTC)" (scheduled at 2025-02-07 04:11:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:11:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:11:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:11:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:37.715878+00:00 (in 41.001863 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:56 UTC)" (scheduled at 2025-02-07 04:11:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:11:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:11:57,468 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:11:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:11:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:08.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:11:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:08 UTC)" (scheduled at 2025-02-07 04:11:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:11:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:11:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 11, 58, 732863), True, datetime.datetime(2025, 2, 7, 4, 11, 58, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:11:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:11:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:11:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:11:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:11:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:14.225378+00:00 (in 14.999550 seconds)
quotatotalworker stdout | 2025-02-07 04:11:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:14 UTC)" (scheduled at 2025-02-07 04:11:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:11:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:11:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:11:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:11:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:11:59,235 [237] [DEBUG] [app] Starting request: urn:request:119228ca-090c-45bd-ac18-d3cc6c040868 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:59,235 [238] [DEBUG] [app] Starting request: urn:request:ee4e465b-f4ff-4664-ba74-7b6710a833d6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:11:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:11:59,238 [246] [DEBUG] [app] Starting request: urn:request:112977f6-6888-4f42-a92f-6cd1cb63f807 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:59,238 [239] [DEBUG] [app] Starting request: urn:request:a5222a94-7889-49c5-b2c2-2eecd5a255a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:11:59,239 [246] [DEBUG] [app] Ending request: urn:request:112977f6-6888-4f42-a92f-6cd1cb63f807 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:112977f6-6888-4f42-a92f-6cd1cb63f807', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:11:59,239 [239] [DEBUG] [app] Ending request: urn:request:a5222a94-7889-49c5-b2c2-2eecd5a255a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5222a94-7889-49c5-b2c2-2eecd5a255a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:11:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:11:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:11:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:11:59,242 [238] [DEBUG] [app] Starting request: urn:request:902d2aa4-6fdc-4704-aa73-75f741a70b3a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:59,242 [238] [DEBUG] [app] Ending request: urn:request:902d2aa4-6fdc-4704-aa73-75f741a70b3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:902d2aa4-6fdc-4704-aa73-75f741a70b3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:11:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:11:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:59,243 [235] [DEBUG] [app] Starting request: urn:request:852e0e04-3ed6-4347-9c97-4c5c268f3090 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:11:59,243 [235] [DEBUG] [app] Ending request: urn:request:852e0e04-3ed6-4347-9c97-4c5c268f3090 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:852e0e04-3ed6-4347-9c97-4c5c268f3090', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:11:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:11:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:11:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:11:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:11:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:11:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:11:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:11:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:11:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:11:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:11:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:11:59,253 [238] [DEBUG] [app] Ending request: urn:request:ee4e465b-f4ff-4664-ba74-7b6710a833d6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee4e465b-f4ff-4664-ba74-7b6710a833d6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 04:11:59,254 [237] [DEBUG] [app] Ending request: urn:request:119228ca-090c-45bd-ac18-d3cc6c040868 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:119228ca-090c-45bd-ac18-d3cc6c040868', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:11:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:11:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:11:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:11:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:11:59,414 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:11:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:11:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:04.936688+00:00 (in 4.998320 seconds)
exportactionlogsworker stdout | 2025-02-07 04:11:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:59 UTC)" (scheduled at 2025-02-07 04:11:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:11:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:11:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:12:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:12:00,540 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:12:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:12:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:23.420886+00:00 (in 20.001264 seconds)
repositorygcworker stdout | 2025-02-07 04:12:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:03 UTC)" (scheduled at 2025-02-07 04:12:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:12:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:12:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 3, 419950), True, datetime.datetime(2025, 2, 7, 4, 12, 3, 419950), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:12:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:12:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:12:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:12:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:12:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:33.919286+00:00 (in 29.999523 seconds)
buildlogsarchiver stdout | 2025-02-07 04:12:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:33 UTC)" (scheduled at 2025-02-07 04:12:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:12:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 12, 3, 920043), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:12:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:12:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:12:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:12:04,778 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:12:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:12:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:59.937899+00:00 (in 55.000702 seconds)
exportactionlogsworker stdout | 2025-02-07 04:12:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:04 UTC)" (scheduled at 2025-02-07 04:12:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:12:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:12:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 4, 937557), True, datetime.datetime(2025, 2, 7, 4, 12, 4, 937557), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:12:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:12:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:12:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:12:05,832 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:12:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:18.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 04:12:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:18 UTC)" (scheduled at 2025-02-07 04:12:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:12:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:12:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 8, 732890), True, datetime.datetime(2025, 2, 7, 4, 12, 8, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:12:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:12:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:12:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:12:09,843 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:12:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:12:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:24.423055+00:00 (in 13.998332 seconds)
gcworker stdout | 2025-02-07 04:12:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:40 UTC)" (scheduled at 2025-02-07 04:12:10.424250+00:00)
gcworker stdout | 2025-02-07 04:12:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:12:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901230431, None, 1, 0])
gcworker stdout | 2025-02-07 04:12:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:12:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:12:10,452 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:12:10,968 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:12:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:12:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:41.831521+00:00 (in 29.999548 seconds)
autopruneworker stdout | 2025-02-07 04:12:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:41 UTC)" (scheduled at 2025-02-07 04:12:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:12:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897931836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:12:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:12:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:12:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:12:12,186 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:12:13,126 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:12:13,146 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
namespacegcworker stdout | 2025-02-07 04:12:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:12:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:31.621131+00:00 (in 17.996380 seconds)
namespacegcworker stdout | 2025-02-07 04:12:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:17:13 UTC)" (scheduled at 2025-02-07 04:12:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:12:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 12, 13, 624993), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:12:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 12, 13, 624993), True, datetime.datetime(2025, 2, 7, 4, 12, 13, 624993), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:12:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 12, 13, 624993), True, datetime.datetime(2025, 2, 7, 4, 12, 13, 624993), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 12, 13, 624993), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:12:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:12:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:17:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:12:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:12:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:29.225378+00:00 (in 14.999430 seconds)
quotatotalworker stdout | 2025-02-07 04:12:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:29 UTC)" (scheduled at 2025-02-07 04:12:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:12:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:12:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
autopruneworker stdout | 2025-02-07 04:12:14,232 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
quotatotalworker stdout | 2025-02-07 04:12:14,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:12:14,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:12:14,236 [237] [DEBUG] [app] Starting request: urn:request:a0ce7ba2-1f83-43b1-a316-50a8be1791d6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:14,236 [235] [DEBUG] [app] Starting request: urn:request:e1752743-c29c-4281-88f3-8007e7cd976e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:14,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:12:14,240 [239] [DEBUG] [app] Starting request: urn:request:da0862a5-17c1-4c53-b0a3-a94cd2e49107 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:14,240 [246] [DEBUG] [app] Starting request: urn:request:204cacb4-ff52-4e83-8b32-0faffab26daa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:14,240 [246] [DEBUG] [app] Ending request: urn:request:204cacb4-ff52-4e83-8b32-0faffab26daa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:204cacb4-ff52-4e83-8b32-0faffab26daa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:12:14,240 [239] [DEBUG] [app] Ending request: urn:request:da0862a5-17c1-4c53-b0a3-a94cd2e49107 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da0862a5-17c1-4c53-b0a3-a94cd2e49107', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:12:14,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:12:14,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:14,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:14,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:14,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:14,244 [237] [DEBUG] [app] Starting request: urn:request:c2f2c0da-b808-4c2b-b5e7-21dfe22b71b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:14,244 [238] [DEBUG] [app] Starting request: urn:request:d61bbed8-ccd4-4914-bfc1-940555ce1cba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:14,244 [237] [DEBUG] [app] Ending request: urn:request:c2f2c0da-b808-4c2b-b5e7-21dfe22b71b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2f2c0da-b808-4c2b-b5e7-21dfe22b71b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:12:14,244 [238] [DEBUG] [app] Ending request: urn:request:d61bbed8-ccd4-4914-bfc1-940555ce1cba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d61bbed8-ccd4-4914-bfc1-940555ce1cba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:12:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:12:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:12:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:14,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:14,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:14,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:14,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:14,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:14,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:14,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:14,257 [237] [DEBUG] [app] Ending request: urn:request:a0ce7ba2-1f83-43b1-a316-50a8be1791d6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0ce7ba2-1f83-43b1-a316-50a8be1791d6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:14,257 [235] [DEBUG] [app] Ending request: urn:request:e1752743-c29c-4281-88f3-8007e7cd976e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1752743-c29c-4281-88f3-8007e7cd976e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:14,257 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:12:14,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:12:14,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:12:14,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:14,645 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:12:15,232 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:12:15,326 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:12:15,445 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:12:15,819 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:12:16,230 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:12:16,353 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:12:16,377 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:12:16,478 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:12:16,481 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:12:16,484 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:12:16,487 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:12:16,489 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:12:17,501 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:12:17,504 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:12:17,508 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:12:18,018 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:12:18,192 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:12:18,471 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:12:18,616 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:12:18,619 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:12:18,622 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:12:18,624 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:12:18,628 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:12:18,631 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:12:18,633 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:12:18,636 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:12:18,638 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:12:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:28.732147+00:00 (in 9.999510 seconds)
notificationworker stdout | 2025-02-07 04:12:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:28 UTC)" (scheduled at 2025-02-07 04:12:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:12:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:12:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 18, 732950), True, datetime.datetime(2025, 2, 7, 4, 12, 18, 732950), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:12:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:12:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:12:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:12:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:12:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:03.419110+00:00 (in 39.997711 seconds)
repositorygcworker stdout | 2025-02-07 04:12:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:23 UTC)" (scheduled at 2025-02-07 04:12:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:12:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:12:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:12:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:12:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:40.424250+00:00 (in 16.000728 seconds)
gcworker stdout | 2025-02-07 04:12:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:54 UTC)" (scheduled at 2025-02-07 04:12:24.423055+00:00)
gcworker stdout | 2025-02-07 04:12:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:12:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:12:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:12:27,504 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:12:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:33.814166+00:00 (in 5.081512 seconds)
notificationworker stdout | 2025-02-07 04:12:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:38 UTC)" (scheduled at 2025-02-07 04:12:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:12:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:12:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 28, 732965), True, datetime.datetime(2025, 2, 7, 4, 12, 28, 732965), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:12:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:12:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:12:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:12:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:12:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:44.225378+00:00 (in 14.999516 seconds)
quotatotalworker stdout | 2025-02-07 04:12:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:44 UTC)" (scheduled at 2025-02-07 04:12:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:12:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:12:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:12:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:12:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:12:29,235 [237] [DEBUG] [app] Starting request: urn:request:9764b101-a8c5-430b-b0a6-f957c553eaa8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:29,235 [235] [DEBUG] [app] Starting request: urn:request:63d36119-28e9-43ad-a4e8-66e29bcda3e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:12:29,238 [246] [DEBUG] [app] Starting request: urn:request:3b2618de-d5c4-4386-bf78-cc53c6895341 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:29,238 [239] [DEBUG] [app] Starting request: urn:request:f5b2614c-b77a-46f2-b312-84d06e8324eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:29,238 [246] [DEBUG] [app] Ending request: urn:request:3b2618de-d5c4-4386-bf78-cc53c6895341 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b2618de-d5c4-4386-bf78-cc53c6895341', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:12:29,239 [239] [DEBUG] [app] Ending request: urn:request:f5b2614c-b77a-46f2-b312-84d06e8324eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5b2614c-b77a-46f2-b312-84d06e8324eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:12:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:12:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:29,242 [238] [DEBUG] [app] Starting request: urn:request:921c9bd1-2ca6-46ef-ab80-5147ea6f539a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:29,242 [236] [DEBUG] [app] Starting request: urn:request:806aa535-c110-4e11-8f9f-0a9ee5be8a57 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:29,242 [236] [DEBUG] [app] Ending request: urn:request:806aa535-c110-4e11-8f9f-0a9ee5be8a57 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:806aa535-c110-4e11-8f9f-0a9ee5be8a57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:12:29,242 [238] [DEBUG] [app] Ending request: urn:request:921c9bd1-2ca6-46ef-ab80-5147ea6f539a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:921c9bd1-2ca6-46ef-ab80-5147ea6f539a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:12:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:12:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:29,254 [235] [DEBUG] [app] Ending request: urn:request:63d36119-28e9-43ad-a4e8-66e29bcda3e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63d36119-28e9-43ad-a4e8-66e29bcda3e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:12:29,254 [237] [DEBUG] [app] Ending request: urn:request:9764b101-a8c5-430b-b0a6-f957c553eaa8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9764b101-a8c5-430b-b0a6-f957c553eaa8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:12:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:12:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:12:29,445 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:12:30,576 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:12:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:12:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:32.625945+00:00 (in 1.004379 seconds)
namespacegcworker stdout | 2025-02-07 04:12:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:31 UTC)" (scheduled at 2025-02-07 04:12:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:12:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:12:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 31, 621888), True, datetime.datetime(2025, 2, 7, 4, 12, 31, 621888), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:12:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:12:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:12:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:12:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:12:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:31.621131+00:00 (in 58.994713 seconds)
namespacegcworker stdout | 2025-02-07 04:12:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:32 UTC)" (scheduled at 2025-02-07 04:12:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:12:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:12:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:12:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:38.732147+00:00 (in 4.917503 seconds)
notificationworker stdout | 2025-02-07 04:12:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:17:33 UTC)" (scheduled at 2025-02-07 04:12:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:12:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 12, 33, 814927), 'notification/%'])
notificationworker stdout | 2025-02-07 04:12:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 12, 33, 814927), True, datetime.datetime(2025, 2, 7, 4, 12, 33, 814927), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:12:33,824 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 12, 33, 814927), True, datetime.datetime(2025, 2, 7, 4, 12, 33, 814927), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 12, 33, 814927), 'notification/%'])
notificationworker stdout | 2025-02-07 04:12:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:12:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:17:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:12:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:12:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:03.919286+00:00 (in 29.999535 seconds)
buildlogsarchiver stdout | 2025-02-07 04:12:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:03 UTC)" (scheduled at 2025-02-07 04:12:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:12:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 12, 33, 920035), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:12:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:12:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:12:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:12:34,810 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:12:35,854 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:56.713547+00:00 (in 18.997172 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:37 UTC)" (scheduled at 2025-02-07 04:12:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:12:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:48.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 04:12:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:48 UTC)" (scheduled at 2025-02-07 04:12:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:12:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:12:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 38, 732924), True, datetime.datetime(2025, 2, 7, 4, 12, 38, 732924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:12:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:12:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:12:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:12:39,854 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:12:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:12:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:54.423055+00:00 (in 13.998389 seconds)
gcworker stdout | 2025-02-07 04:12:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:10 UTC)" (scheduled at 2025-02-07 04:12:40.424250+00:00)
gcworker stdout | 2025-02-07 04:12:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:12:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901260431, None, 1, 0])
gcworker stdout | 2025-02-07 04:12:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:12:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:12:40,488 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:12:41,004 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:12:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:12:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:11.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 04:12:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:11 UTC)" (scheduled at 2025-02-07 04:12:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:12:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897961836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:12:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:12:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:12:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:12:42,222 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:12:43,162 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:12:43,182 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:12:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:12:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:59.225378+00:00 (in 14.999431 seconds)
quotatotalworker stdout | 2025-02-07 04:12:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:59 UTC)" (scheduled at 2025-02-07 04:12:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:12:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:12:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:12:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:12:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:12:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:12:44,236 [237] [DEBUG] [app] Starting request: urn:request:63340bd0-4eb3-473b-8f05-b7dec4452fff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:44,236 [235] [DEBUG] [app] Starting request: urn:request:20f3a6af-a688-4cde-ac4e-4d0339d58b90 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:12:44,239 [239] [DEBUG] [app] Starting request: urn:request:1d870c4e-8301-4861-ba11-a4c463483f58 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:44,239 [245] [DEBUG] [app] Starting request: urn:request:866d162d-b231-42a6-991d-bc3d4559ff53 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:44,239 [245] [DEBUG] [app] Ending request: urn:request:866d162d-b231-42a6-991d-bc3d4559ff53 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:866d162d-b231-42a6-991d-bc3d4559ff53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:12:44,239 [239] [DEBUG] [app] Ending request: urn:request:1d870c4e-8301-4861-ba11-a4c463483f58 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1d870c4e-8301-4861-ba11-a4c463483f58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:12:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:12:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:44,242 [236] [DEBUG] [app] Starting request: urn:request:7ed259da-b581-42dd-b635-75ea16b5d300 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:44,242 [238] [DEBUG] [app] Starting request: urn:request:b3acbbc0-07d8-4fe0-85e7-d56b4d04877c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:44,243 [236] [DEBUG] [app] Ending request: urn:request:7ed259da-b581-42dd-b635-75ea16b5d300 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ed259da-b581-42dd-b635-75ea16b5d300', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:12:44,243 [238] [DEBUG] [app] Ending request: urn:request:b3acbbc0-07d8-4fe0-85e7-d56b4d04877c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3acbbc0-07d8-4fe0-85e7-d56b4d04877c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:12:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:44,254 [235] [DEBUG] [app] Ending request: urn:request:20f3a6af-a688-4cde-ac4e-4d0339d58b90 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20f3a6af-a688-4cde-ac4e-4d0339d58b90', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:12:44,254 [237] [DEBUG] [app] Ending request: urn:request:63340bd0-4eb3-473b-8f05-b7dec4452fff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63340bd0-4eb3-473b-8f05-b7dec4452fff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:12:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:12:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:12:44,271 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:44,681 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:12:45,268 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:12:45,359 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:12:45,481 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:12:45,831 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:12:46,248 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:12:46,389 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:12:46,413 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:12:46,486 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:12:46,489 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:12:46,493 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:12:46,495 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:12:46,498 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:12:47,510 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:12:47,513 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:12:47,515 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:12:48,054 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:12:48,217 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:12:48,495 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:12:48,626 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:12:48,628 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:12:48,632 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:12:48,636 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:12:48,639 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:12:48,642 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:12:48,645 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:12:48,648 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:12:48,651 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:12:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:50.814677+00:00 (in 2.082049 seconds)
notificationworker stdout | 2025-02-07 04:12:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:58 UTC)" (scheduled at 2025-02-07 04:12:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:12:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:12:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 48, 732910), True, datetime.datetime(2025, 2, 7, 4, 12, 48, 732910), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:12:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:12:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:12:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:12:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:12:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:12:58.732147+00:00 (in 7.917013 seconds)
notificationworker stdout | 2025-02-07 04:12:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:50 UTC)" (scheduled at 2025-02-07 04:12:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:12:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:12:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:12:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:12:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:10.424250+00:00 (in 16.000713 seconds)
gcworker stdout | 2025-02-07 04:12:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:24 UTC)" (scheduled at 2025-02-07 04:12:54.423055+00:00)
gcworker stdout | 2025-02-07 04:12:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:12:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:12:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:12:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:56.118855+00:00 (in 59.999505 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:12:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:56 UTC)" (scheduled at 2025-02-07 04:12:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:12:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:12:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:12:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:37.715878+00:00 (in 41.001882 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:56 UTC)" (scheduled at 2025-02-07 04:12:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:12:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:12:57,537 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:12:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:12:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:08.732147+00:00 (in 9.999524 seconds)
notificationworker stdout | 2025-02-07 04:12:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:08 UTC)" (scheduled at 2025-02-07 04:12:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:12:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:12:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 12, 58, 732902), True, datetime.datetime(2025, 2, 7, 4, 12, 58, 732902), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:12:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:12:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:12:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:12:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:12:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:14.225378+00:00 (in 14.999521 seconds)
quotatotalworker stdout | 2025-02-07 04:12:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:14 UTC)" (scheduled at 2025-02-07 04:12:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:12:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:12:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:12:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:12:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:12:59,235 [238] [DEBUG] [app] Starting request: urn:request:16c13a1a-851e-4519-80bf-857f3ceeb3e8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:59,235 [235] [DEBUG] [app] Starting request: urn:request:cf1dbfb8-a9b4-4ef5-b330-edc89bc13b5f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:12:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:12:59,238 [246] [DEBUG] [app] Starting request: urn:request:290e83d1-3af4-481c-8488-2ba8ed235378 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:59,238 [239] [DEBUG] [app] Starting request: urn:request:93b72678-0c03-47f6-ba50-5342b6927cd3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:12:59,238 [246] [DEBUG] [app] Ending request: urn:request:290e83d1-3af4-481c-8488-2ba8ed235378 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:290e83d1-3af4-481c-8488-2ba8ed235378', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:12:59,238 [239] [DEBUG] [app] Ending request: urn:request:93b72678-0c03-47f6-ba50-5342b6927cd3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93b72678-0c03-47f6-ba50-5342b6927cd3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:12:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:12:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:12:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:12:59,241 [237] [DEBUG] [app] Starting request: urn:request:e4a48124-dce7-4fe9-9b85-4591e1648ae1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:59,242 [237] [DEBUG] [app] Ending request: urn:request:e4a48124-dce7-4fe9-9b85-4591e1648ae1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4a48124-dce7-4fe9-9b85-4591e1648ae1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:12:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:12:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:59,242 [236] [DEBUG] [app] Starting request: urn:request:4df2b4cd-28d0-478b-ada5-7a8809937263 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:12:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:59,242 [236] [DEBUG] [app] Ending request: urn:request:4df2b4cd-28d0-478b-ada5-7a8809937263 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4df2b4cd-28d0-478b-ada5-7a8809937263', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:12:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:12:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:12:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:12:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:12:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:12:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:12:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:12:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:12:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:12:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:12:59,253 [235] [DEBUG] [app] Ending request: urn:request:cf1dbfb8-a9b4-4ef5-b330-edc89bc13b5f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf1dbfb8-a9b4-4ef5-b330-edc89bc13b5f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:12:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 04:12:59,254 [238] [DEBUG] [app] Ending request: urn:request:16c13a1a-851e-4519-80bf-857f3ceeb3e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16c13a1a-851e-4519-80bf-857f3ceeb3e8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:12:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:12:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:12:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:12:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:12:59,458 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:12:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:12:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:04.936688+00:00 (in 4.998256 seconds)
exportactionlogsworker stdout | 2025-02-07 04:12:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:59 UTC)" (scheduled at 2025-02-07 04:12:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:12:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:12:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:13:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:13:00,588 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:13:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:13:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:23.420886+00:00 (in 20.001293 seconds)
repositorygcworker stdout | 2025-02-07 04:13:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:03 UTC)" (scheduled at 2025-02-07 04:13:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:13:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:13:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 3, 419871), True, datetime.datetime(2025, 2, 7, 4, 13, 3, 419871), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:13:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:13:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:13:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:13:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:13:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:33.919286+00:00 (in 29.999526 seconds)
buildlogsarchiver stdout | 2025-02-07 04:13:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:33 UTC)" (scheduled at 2025-02-07 04:13:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:13:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 13, 3, 920032), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:13:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:13:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:13:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:13:04,843 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:13:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:13:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:59.937899+00:00 (in 55.000782 seconds)
exportactionlogsworker stdout | 2025-02-07 04:13:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:04 UTC)" (scheduled at 2025-02-07 04:13:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:13:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:13:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 4, 937393), True, datetime.datetime(2025, 2, 7, 4, 13, 4, 937393), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:13:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:13:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:13:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:13:05,890 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:13:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:13:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:18.732147+00:00 (in 9.999524 seconds)
notificationworker stdout | 2025-02-07 04:13:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:18 UTC)" (scheduled at 2025-02-07 04:13:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:13:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:13:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 8, 732913), True, datetime.datetime(2025, 2, 7, 4, 13, 8, 732913), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:13:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:13:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:13:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:13:09,860 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:13:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:13:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:24.423055+00:00 (in 13.998312 seconds)
gcworker stdout | 2025-02-07 04:13:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:40 UTC)" (scheduled at 2025-02-07 04:13:10.424250+00:00)
gcworker stdout | 2025-02-07 04:13:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:13:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901290431, None, 1, 0])
gcworker stdout | 2025-02-07 04:13:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:13:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:13:10,522 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:13:11,018 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:13:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:13:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:41.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 04:13:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:41 UTC)" (scheduled at 2025-02-07 04:13:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:13:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738897991836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:13:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:13:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:13:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:13:12,258 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:13:13,198 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:13:13,208 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:13:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:13:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:29.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 04:13:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:29 UTC)" (scheduled at 2025-02-07 04:13:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:13:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:13:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:13:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:13:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:13:14,235 [237] [DEBUG] [app] Starting request: urn:request:cfe87aef-71b6-4050-9540-4a926db0d3d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:14,235 [238] [DEBUG] [app] Starting request: urn:request:b499dbbb-40db-4796-8ca8-2c000d3e64fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:13:14,238 [239] [DEBUG] [app] Starting request: urn:request:6cb33436-988d-4ac8-b05a-7c1cf5999cd7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:14,238 [245] [DEBUG] [app] Starting request: urn:request:8a72aef9-9f4f-4745-9418-a11d733db0e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:14,239 [239] [DEBUG] [app] Ending request: urn:request:6cb33436-988d-4ac8-b05a-7c1cf5999cd7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6cb33436-988d-4ac8-b05a-7c1cf5999cd7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:13:14,239 [245] [DEBUG] [app] Ending request: urn:request:8a72aef9-9f4f-4745-9418-a11d733db0e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a72aef9-9f4f-4745-9418-a11d733db0e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:13:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:13:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:13:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:14,241 [236] [DEBUG] [app] Starting request: urn:request:94d8b162-7a18-4b40-b08c-60a5e60de654 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:14,242 [236] [DEBUG] [app] Ending request: urn:request:94d8b162-7a18-4b40-b08c-60a5e60de654 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:94d8b162-7a18-4b40-b08c-60a5e60de654', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:13:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:14,242 [235] [DEBUG] [app] Starting request: urn:request:4a3607a7-a109-453c-8a11-11ca6e9b3dd3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:13:14,243 [235] [DEBUG] [app] Ending request: urn:request:4a3607a7-a109-453c-8a11-11ca6e9b3dd3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a3607a7-a109-453c-8a11-11ca6e9b3dd3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:13:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:13:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:14,253 [238] [DEBUG] [app] Ending request: urn:request:b499dbbb-40db-4796-8ca8-2c000d3e64fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b499dbbb-40db-4796-8ca8-2c000d3e64fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:13:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:13:14,254 [237] [DEBUG] [app] Ending request: urn:request:cfe87aef-71b6-4050-9540-4a926db0d3d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cfe87aef-71b6-4050-9540-4a926db0d3d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:13:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
autopruneworker stdout | 2025-02-07 04:13:14,292 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:14,717 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:13:15,292 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:13:15,375 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:13:15,518 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:13:15,850 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:13:16,282 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:13:16,401 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:13:16,442 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:13:16,494 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:13:16,499 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:13:16,502 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:13:16,505 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:13:16,508 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:13:17,519 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:13:17,523 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:13:17,526 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:13:18,078 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:13:18,234 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:13:18,511 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:13:18,635 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:13:18,637 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:13:18,640 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:13:18,643 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:13:18,650 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:13:18,653 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:13:18,656 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:13:18,659 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:13:18,662 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:13:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:13:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 04:13:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:28 UTC)" (scheduled at 2025-02-07 04:13:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:13:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:13:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 18, 732871), True, datetime.datetime(2025, 2, 7, 4, 13, 18, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:13:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:13:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:13:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:13:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:13:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:32.420707+00:00 (in 8.999319 seconds)
repositorygcworker stdout | 2025-02-07 04:13:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:23 UTC)" (scheduled at 2025-02-07 04:13:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:13:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:13:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:13:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:13:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:40.424250+00:00 (in 16.000744 seconds)
gcworker stdout | 2025-02-07 04:13:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:54 UTC)" (scheduled at 2025-02-07 04:13:24.423055+00:00)
gcworker stdout | 2025-02-07 04:13:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:13:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:13:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:13:27,573 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:13:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:13:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:38.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 04:13:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:38 UTC)" (scheduled at 2025-02-07 04:13:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:13:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:13:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 28, 732858), True, datetime.datetime(2025, 2, 7, 4, 13, 28, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:13:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:13:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:13:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:13:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:13:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:44.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 04:13:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:44 UTC)" (scheduled at 2025-02-07 04:13:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:13:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:13:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:13:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:13:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:13:29,235 [235] [DEBUG] [app] Starting request: urn:request:10b5b2ba-47fb-4425-a789-523a4b462ad6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:29,235 [238] [DEBUG] [app] Starting request: urn:request:32a4faa9-e6c8-42f2-a692-4af72a358064 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:13:29,238 [239] [DEBUG] [app] Starting request: urn:request:963461a5-fa27-4ec1-b918-92dedd5d236f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:29,239 [246] [DEBUG] [app] Starting request: urn:request:0a8c13da-50bf-4df5-9a61-247bf47d252e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:29,239 [239] [DEBUG] [app] Ending request: urn:request:963461a5-fa27-4ec1-b918-92dedd5d236f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:963461a5-fa27-4ec1-b918-92dedd5d236f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:13:29,239 [246] [DEBUG] [app] Ending request: urn:request:0a8c13da-50bf-4df5-9a61-247bf47d252e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a8c13da-50bf-4df5-9a61-247bf47d252e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:13:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:13:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:29,242 [236] [DEBUG] [app] Starting request: urn:request:d58d9ab3-bfaa-4508-ab3c-d6e6516d6fbb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:29,242 [235] [DEBUG] [app] Starting request: urn:request:bb99a890-a744-40db-b46e-9c4b2e6a2ca8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:29,242 [236] [DEBUG] [app] Ending request: urn:request:d58d9ab3-bfaa-4508-ab3c-d6e6516d6fbb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d58d9ab3-bfaa-4508-ab3c-d6e6516d6fbb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:13:29,242 [235] [DEBUG] [app] Ending request: urn:request:bb99a890-a744-40db-b46e-9c4b2e6a2ca8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb99a890-a744-40db-b46e-9c4b2e6a2ca8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:13:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:13:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:13:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:13:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:29,253 [235] [DEBUG] [app] Ending request: urn:request:10b5b2ba-47fb-4425-a789-523a4b462ad6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10b5b2ba-47fb-4425-a789-523a4b462ad6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:29,253 [238] [DEBUG] [app] Ending request: urn:request:32a4faa9-e6c8-42f2-a692-4af72a358064 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:32a4faa9-e6c8-42f2-a692-4af72a358064', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:13:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:13:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:13:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:13:29,473 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:13:30,623 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:13:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:13:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:32.625945+00:00 (in 1.004356 seconds)
namespacegcworker stdout | 2025-02-07 04:13:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:31 UTC)" (scheduled at 2025-02-07 04:13:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:13:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:13:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 31, 621863), True, datetime.datetime(2025, 2, 7, 4, 13, 31, 621863), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:13:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:13:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:13:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:13:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:13:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:03.419110+00:00 (in 30.997932 seconds)
repositorygcworker stdout | 2025-02-07 04:13:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:18:32 UTC)" (scheduled at 2025-02-07 04:13:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:13:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 13, 32, 421500), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:13:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 13, 32, 421500), True, datetime.datetime(2025, 2, 7, 4, 13, 32, 421500), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:13:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 13, 32, 421500), True, datetime.datetime(2025, 2, 7, 4, 13, 32, 421500), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 13, 32, 421500), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:13:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:13:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:18:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:13:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:13:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:31.621131+00:00 (in 58.994727 seconds)
namespacegcworker stdout | 2025-02-07 04:13:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:32 UTC)" (scheduled at 2025-02-07 04:13:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:13:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:13:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:13:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:13:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:03.919286+00:00 (in 29.999539 seconds)
buildlogsarchiver stdout | 2025-02-07 04:13:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:03 UTC)" (scheduled at 2025-02-07 04:13:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:13:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 13, 33, 920029), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:13:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:13:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:13:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:13:34,879 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:13:35,926 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:56.713547+00:00 (in 18.997212 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:37 UTC)" (scheduled at 2025-02-07 04:13:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:13:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:13:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:48.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:13:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:48 UTC)" (scheduled at 2025-02-07 04:13:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:13:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:13:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 38, 732898), True, datetime.datetime(2025, 2, 7, 4, 13, 38, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:13:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:13:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:13:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:13:39,896 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:13:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:13:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:54.423055+00:00 (in 13.998333 seconds)
gcworker stdout | 2025-02-07 04:13:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:10 UTC)" (scheduled at 2025-02-07 04:13:40.424250+00:00)
gcworker stdout | 2025-02-07 04:13:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:13:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901320431, None, 1, 0])
gcworker stdout | 2025-02-07 04:13:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:13:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:13:40,542 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:13:41,031 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:13:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:13:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:11.831521+00:00 (in 29.999552 seconds)
autopruneworker stdout | 2025-02-07 04:13:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:11 UTC)" (scheduled at 2025-02-07 04:13:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:13:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898021836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:13:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:13:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:13:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:13:42,281 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:13:43,209 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:13:43,248 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:13:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:13:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:59.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 04:13:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:59 UTC)" (scheduled at 2025-02-07 04:13:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:13:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:13:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:13:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:13:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:13:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:13:44,235 [235] [DEBUG] [app] Starting request: urn:request:a53b1442-72d9-4e68-8437-78057a0fd931 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:44,235 [238] [DEBUG] [app] Starting request: urn:request:0aa467f4-8b39-45b8-b406-5eda72d08d05 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:13:44,238 [246] [DEBUG] [app] Starting request: urn:request:079f0338-34ec-4106-9ed1-b14cd248bbf6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:44,238 [245] [DEBUG] [app] Starting request: urn:request:1ab54b6b-d2dd-4bd8-af8d-7b9fa52fc4c9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:44,239 [245] [DEBUG] [app] Ending request: urn:request:1ab54b6b-d2dd-4bd8-af8d-7b9fa52fc4c9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1ab54b6b-d2dd-4bd8-af8d-7b9fa52fc4c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:13:44,239 [246] [DEBUG] [app] Ending request: urn:request:079f0338-34ec-4106-9ed1-b14cd248bbf6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:079f0338-34ec-4106-9ed1-b14cd248bbf6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:13:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:13:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:44,241 [235] [DEBUG] [app] Starting request: urn:request:798d6f8a-3d1a-41d8-bd87-d15ad57ad42a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:44,242 [235] [DEBUG] [app] Ending request: urn:request:798d6f8a-3d1a-41d8-bd87-d15ad57ad42a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:798d6f8a-3d1a-41d8-bd87-d15ad57ad42a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:13:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:44,242 [237] [DEBUG] [app] Starting request: urn:request:7a8b3425-8e76-4070-9196-c0afd8801666 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:44,243 [237] [DEBUG] [app] Ending request: urn:request:7a8b3425-8e76-4070-9196-c0afd8801666 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a8b3425-8e76-4070-9196-c0afd8801666', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:13:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:13:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:13:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:13:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:44,253 [235] [DEBUG] [app] Ending request: urn:request:a53b1442-72d9-4e68-8437-78057a0fd931 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a53b1442-72d9-4e68-8437-78057a0fd931', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:13:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:13:44,254 [238] [DEBUG] [app] Ending request: urn:request:0aa467f4-8b39-45b8-b406-5eda72d08d05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0aa467f4-8b39-45b8-b406-5eda72d08d05', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:13:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
autopruneworker stdout | 2025-02-07 04:13:44,328 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:44,736 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:13:45,315 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:13:45,386 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:13:45,554 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:13:45,873 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:13:46,318 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:13:46,415 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:13:46,466 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:13:46,503 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:13:46,508 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:13:46,512 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:13:46,515 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:13:46,517 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:13:47,527 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:13:47,530 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:13:47,533 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:13:48,112 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:13:48,241 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:13:48,547 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:13:48,643 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:13:48,650 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:13:48,654 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:13:48,658 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:13:48,662 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:13:48,665 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:13:48,668 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:13:48,671 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:13:48,675 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:13:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:13:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:50.814677+00:00 (in 2.082053 seconds)
notificationworker stdout | 2025-02-07 04:13:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:58 UTC)" (scheduled at 2025-02-07 04:13:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:13:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:13:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 48, 732926), True, datetime.datetime(2025, 2, 7, 4, 13, 48, 732926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:13:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:13:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:13:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:13:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:13:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:13:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:13:58.732147+00:00 (in 7.917028 seconds)
notificationworker stdout | 2025-02-07 04:13:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:50 UTC)" (scheduled at 2025-02-07 04:13:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:13:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:13:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:13:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:13:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:10.424250+00:00 (in 16.000706 seconds)
gcworker stdout | 2025-02-07 04:13:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:24 UTC)" (scheduled at 2025-02-07 04:13:54.423055+00:00)
gcworker stdout | 2025-02-07 04:13:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:13:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:13:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:13:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:56.118855+00:00 (in 59.999496 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:13:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:56 UTC)" (scheduled at 2025-02-07 04:13:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:13:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:13:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:13:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:37.715878+00:00 (in 41.001860 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:56 UTC)" (scheduled at 2025-02-07 04:13:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:13:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:13:57,581 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:13:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:13:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:08.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 04:13:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:08 UTC)" (scheduled at 2025-02-07 04:13:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:13:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:13:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 13, 58, 732866), True, datetime.datetime(2025, 2, 7, 4, 13, 58, 732866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:13:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:13:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:13:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:13:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:13:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:14.225378+00:00 (in 14.999575 seconds)
quotatotalworker stdout | 2025-02-07 04:13:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:14 UTC)" (scheduled at 2025-02-07 04:13:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:13:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:13:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:13:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:13:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:13:59,236 [235] [DEBUG] [app] Starting request: urn:request:c86085e9-52e7-48d6-85ee-9328bb285122 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:59,236 [238] [DEBUG] [app] Starting request: urn:request:76176f53-8ae0-448c-9a66-11f2e7d988d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:13:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:13:59,238 [245] [DEBUG] [app] Starting request: urn:request:5350bf60-b9b3-4503-8bf9-117f9f668fe7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:59,238 [239] [DEBUG] [app] Starting request: urn:request:d2ad3628-645f-442b-b414-3b30c4e5fa92 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:13:59,239 [239] [DEBUG] [app] Ending request: urn:request:d2ad3628-645f-442b-b414-3b30c4e5fa92 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2ad3628-645f-442b-b414-3b30c4e5fa92', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:13:59,239 [245] [DEBUG] [app] Ending request: urn:request:5350bf60-b9b3-4503-8bf9-117f9f668fe7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5350bf60-b9b3-4503-8bf9-117f9f668fe7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:13:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:13:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:13:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:13:59,241 [235] [DEBUG] [app] Starting request: urn:request:129568ea-84cb-469b-9bbb-695095b5b525 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:59,242 [235] [DEBUG] [app] Ending request: urn:request:129568ea-84cb-469b-9bbb-695095b5b525 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:129568ea-84cb-469b-9bbb-695095b5b525', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:13:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:13:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:59,242 [237] [DEBUG] [app] Starting request: urn:request:a54cc146-7d6f-4dbf-8ea8-4823be381871 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:13:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:59,243 [237] [DEBUG] [app] Ending request: urn:request:a54cc146-7d6f-4dbf-8ea8-4823be381871 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a54cc146-7d6f-4dbf-8ea8-4823be381871', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:13:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:13:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:13:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:13:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:13:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:13:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:13:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:13:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:13:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:13:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:13:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:13:59,253 [235] [DEBUG] [app] Ending request: urn:request:c86085e9-52e7-48d6-85ee-9328bb285122 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c86085e9-52e7-48d6-85ee-9328bb285122', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:13:59,253 [238] [DEBUG] [app] Ending request: urn:request:76176f53-8ae0-448c-9a66-11f2e7d988d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76176f53-8ae0-448c-9a66-11f2e7d988d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:13:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
gunicorn-web stdout | 2025-02-07 04:13:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:13:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:13:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:13:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:13:59,492 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:13:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:13:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:04.936688+00:00 (in 4.998341 seconds)
exportactionlogsworker stdout | 2025-02-07 04:13:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:59 UTC)" (scheduled at 2025-02-07 04:13:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:13:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:13:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:14:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:14:00,634 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:14:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:14:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:23.420886+00:00 (in 20.001260 seconds)
repositorygcworker stdout | 2025-02-07 04:14:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:03 UTC)" (scheduled at 2025-02-07 04:14:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:14:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:14:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 3, 419911), True, datetime.datetime(2025, 2, 7, 4, 14, 3, 419911), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:14:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:14:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:14:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:14:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:14:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:33.919286+00:00 (in 29.999514 seconds)
buildlogsarchiver stdout | 2025-02-07 04:14:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:33 UTC)" (scheduled at 2025-02-07 04:14:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:14:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 14, 3, 920027), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:14:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:14:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:14:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:33 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:14:04,907 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
exportactionlogsworker stdout | 2025-02-07 04:14:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:14:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:52.937648+00:00 (in 48.000515 seconds)
exportactionlogsworker stdout | 2025-02-07 04:14:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:04 UTC)" (scheduled at 2025-02-07 04:14:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:14:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:14:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 4, 937436), True, datetime.datetime(2025, 2, 7, 4, 14, 4, 937436), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:14:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:14:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:14:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:04 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:14:05,953 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:14:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:14:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:18.732147+00:00 (in 9.999547 seconds)
notificationworker stdout | 2025-02-07 04:14:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:18 UTC)" (scheduled at 2025-02-07 04:14:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:14:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:14:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 8, 732890), True, datetime.datetime(2025, 2, 7, 4, 14, 8, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:14:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:14:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:14:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:14:09,907 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:14:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:14:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:24.423055+00:00 (in 13.998375 seconds)
gcworker stdout | 2025-02-07 04:14:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:40 UTC)" (scheduled at 2025-02-07 04:14:10.424250+00:00)
gcworker stdout | 2025-02-07 04:14:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:14:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901350431, None, 1, 0])
gcworker stdout | 2025-02-07 04:14:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:14:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:14:10,569 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:14:11,066 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:14:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:14:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:41.831521+00:00 (in 29.999565 seconds)
autopruneworker stdout | 2025-02-07 04:14:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:41 UTC)" (scheduled at 2025-02-07 04:14:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:14:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898051836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:14:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:14:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:14:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:14:12,317 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:14:13,242 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:14:13,285 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:14:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:14:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:29.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 04:14:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:29 UTC)" (scheduled at 2025-02-07 04:14:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:14:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:14:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:14:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:14:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:14:14,235 [235] [DEBUG] [app] Starting request: urn:request:9ba7bf93-b03c-4a37-9346-07937cdf4fca (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:14,235 [238] [DEBUG] [app] Starting request: urn:request:f07ad6c5-3de8-4c1a-bb55-06870e0c8d38 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:14:14,239 [246] [DEBUG] [app] Starting request: urn:request:a9668cb5-5d07-46b9-bb33-5ed4596aaf26 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:14:14,239 [245] [DEBUG] [app] Starting request: urn:request:7f96bcc6-9f56-444e-9f26-630db2c2b94b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:14:14,239 [246] [DEBUG] [app] Ending request: urn:request:a9668cb5-5d07-46b9-bb33-5ed4596aaf26 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9668cb5-5d07-46b9-bb33-5ed4596aaf26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:14:14,239 [245] [DEBUG] [app] Ending request: urn:request:7f96bcc6-9f56-444e-9f26-630db2c2b94b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f96bcc6-9f56-444e-9f26-630db2c2b94b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:14:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:14:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:14:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:14,242 [235] [DEBUG] [app] Starting request: urn:request:92a7165f-5e06-465a-b6ad-120003d384bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:14,242 [235] [DEBUG] [app] Ending request: urn:request:92a7165f-5e06-465a-b6ad-120003d384bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92a7165f-5e06-465a-b6ad-120003d384bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:14:14,243 [236] [DEBUG] [app] Starting request: urn:request:3ec719e4-5653-48b4-8b53-74fe1bccb722 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:14,243 [236] [DEBUG] [app] Ending request: urn:request:3ec719e4-5653-48b4-8b53-74fe1bccb722 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ec719e4-5653-48b4-8b53-74fe1bccb722', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:14:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:14:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:14:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:14:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:14,255 [235] [DEBUG] [app] Ending request: urn:request:9ba7bf93-b03c-4a37-9346-07937cdf4fca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ba7bf93-b03c-4a37-9346-07937cdf4fca', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:14:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:14:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:14:14,255 [238] [DEBUG] [app] Ending request: urn:request:f07ad6c5-3de8-4c1a-bb55-06870e0c8d38 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f07ad6c5-3de8-4c1a-bb55-06870e0c8d38', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:14:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
gunicorn-web stdout | 2025-02-07 04:14:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:14:14,363 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:14,772 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:14:15,353 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:14:15,415 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:14:15,579 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:14:15,902 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:14:16,344 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:14:16,452 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:14:16,477 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:14:16,511 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:14:16,516 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:14:16,520 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:14:16,523 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:14:16,525 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:14:17,535 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:14:17,538 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:14:17,541 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:14:18,149 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:14:18,277 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:14:18,581 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:14:18,655 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:14:18,660 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:14:18,665 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:14:18,667 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:14:18,671 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:14:18,673 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:14:18,677 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:14:18,681 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:14:18,684 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:14:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:14:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:28.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 04:14:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:28 UTC)" (scheduled at 2025-02-07 04:14:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:14:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:14:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 18, 732875), True, datetime.datetime(2025, 2, 7, 4, 14, 18, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:14:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:14:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:14:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:14:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:14:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:03.419110+00:00 (in 39.997760 seconds)
repositorygcworker stdout | 2025-02-07 04:14:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:23 UTC)" (scheduled at 2025-02-07 04:14:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:14:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:14:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:14:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:14:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:40.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 04:14:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:54 UTC)" (scheduled at 2025-02-07 04:14:24.423055+00:00)
gcworker stdout | 2025-02-07 04:14:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:14:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:14:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:14:27,617 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:14:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:14:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:38.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 04:14:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:38 UTC)" (scheduled at 2025-02-07 04:14:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:14:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:14:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 28, 732873), True, datetime.datetime(2025, 2, 7, 4, 14, 28, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:14:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:14:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:14:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:14:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:14:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:44.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:14:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:44 UTC)" (scheduled at 2025-02-07 04:14:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:14:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:14:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:14:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:14:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:14:29,236 [237] [DEBUG] [app] Starting request: urn:request:b9b2acd5-33a2-49ac-88cf-a18140d5a261 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:29,236 [235] [DEBUG] [app] Starting request: urn:request:a037791f-68ec-4225-a1dc-3a6ef56e5eea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:14:29,239 [239] [DEBUG] [app] Starting request: urn:request:2bcd64e3-afd9-49ba-ac70-620a0d08c2c7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:14:29,239 [239] [DEBUG] [app] Ending request: urn:request:2bcd64e3-afd9-49ba-ac70-620a0d08c2c7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2bcd64e3-afd9-49ba-ac70-620a0d08c2c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:14:29,239 [245] [DEBUG] [app] Starting request: urn:request:fab606c4-f3b2-47be-9155-1fdfad9aa13e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:14:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:14:29,240 [245] [DEBUG] [app] Ending request: urn:request:fab606c4-f3b2-47be-9155-1fdfad9aa13e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fab606c4-f3b2-47be-9155-1fdfad9aa13e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:14:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:14:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:29,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:29,243 [235] [DEBUG] [app] Starting request: urn:request:1e302d65-04a8-4c7c-b491-f39b284ac96e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:29,243 [235] [DEBUG] [app] Ending request: urn:request:1e302d65-04a8-4c7c-b491-f39b284ac96e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e302d65-04a8-4c7c-b491-f39b284ac96e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:14:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:29,244 [238] [DEBUG] [app] Starting request: urn:request:567099e1-9ea3-4679-b6a8-5c65492ff795 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:14:29,244 [238] [DEBUG] [app] Ending request: urn:request:567099e1-9ea3-4679-b6a8-5c65492ff795 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:567099e1-9ea3-4679-b6a8-5c65492ff795', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:29,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:14:29,245 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:29,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:29,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:14:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:29,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:29,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:29,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:29,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:29,255 [235] [DEBUG] [app] Ending request: urn:request:a037791f-68ec-4225-a1dc-3a6ef56e5eea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a037791f-68ec-4225-a1dc-3a6ef56e5eea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:14:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:14:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:14:29,256 [237] [DEBUG] [app] Ending request: urn:request:b9b2acd5-33a2-49ac-88cf-a18140d5a261 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b9b2acd5-33a2-49ac-88cf-a18140d5a261', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:14:29,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:14:29,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
repositoryactioncounter stdout | 2025-02-07 04:14:29,506 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:14:30,651 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:14:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:14:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:32.625945+00:00 (in 1.004373 seconds)
namespacegcworker stdout | 2025-02-07 04:14:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:31 UTC)" (scheduled at 2025-02-07 04:14:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:14:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:14:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 31, 621886), True, datetime.datetime(2025, 2, 7, 4, 14, 31, 621886), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:14:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:14:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:14:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:14:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:14:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:31.621131+00:00 (in 58.994655 seconds)
namespacegcworker stdout | 2025-02-07 04:14:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:32 UTC)" (scheduled at 2025-02-07 04:14:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:14:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:14:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:14:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:14:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:03.919286+00:00 (in 29.999497 seconds)
buildlogsarchiver stdout | 2025-02-07 04:14:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:03 UTC)" (scheduled at 2025-02-07 04:14:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:14:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 14, 33, 920017), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:14:33,930 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:14:33,930 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:14:33,930 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:14:34,943 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:14:35,989 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:56.713547+00:00 (in 18.997162 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:37 UTC)" (scheduled at 2025-02-07 04:14:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:14:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:14:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:48.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:14:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:48 UTC)" (scheduled at 2025-02-07 04:14:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:14:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:14:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 38, 732918), True, datetime.datetime(2025, 2, 7, 4, 14, 38, 732918), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:14:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:14:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:14:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:14:39,915 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:14:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:14:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:54.423055+00:00 (in 13.998367 seconds)
gcworker stdout | 2025-02-07 04:14:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:10 UTC)" (scheduled at 2025-02-07 04:14:40.424250+00:00)
gcworker stdout | 2025-02-07 04:14:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:14:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901380431, None, 1, 0])
gcworker stdout | 2025-02-07 04:14:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:14:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:14:40,589 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:14:41,077 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:14:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:14:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:11.831521+00:00 (in 29.999544 seconds)
autopruneworker stdout | 2025-02-07 04:14:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:11 UTC)" (scheduled at 2025-02-07 04:14:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:14:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898081836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:14:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:14:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:14:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:14:42,354 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:14:43,275 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:14:43,321 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:14:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:14:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:59.225378+00:00 (in 14.999403 seconds)
quotatotalworker stdout | 2025-02-07 04:14:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:59 UTC)" (scheduled at 2025-02-07 04:14:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:14:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:14:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:14:44,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:14:44,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:14:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:14:44,236 [235] [DEBUG] [app] Starting request: urn:request:e665fb6c-8e4d-4536-8af2-611fea9b6a1e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:44,236 [237] [DEBUG] [app] Starting request: urn:request:da923caa-8836-4fc1-b9ac-d2d3788c2801 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:14:44,239 [245] [DEBUG] [app] Starting request: urn:request:59024297-80fc-400a-a07f-3d739a374561 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:14:44,239 [245] [DEBUG] [app] Ending request: urn:request:59024297-80fc-400a-a07f-3d739a374561 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59024297-80fc-400a-a07f-3d739a374561', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:14:44,239 [241] [DEBUG] [app] Starting request: urn:request:916467bc-1dc4-4029-95a2-d38080a83390 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:14:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:14:44,239 [241] [DEBUG] [app] Ending request: urn:request:916467bc-1dc4-4029-95a2-d38080a83390 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:916467bc-1dc4-4029-95a2-d38080a83390', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:14:44,240 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:14:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:44,242 [235] [DEBUG] [app] Starting request: urn:request:1ccaffd5-a06d-44c6-afe9-f7cd84df3b6d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:44,242 [235] [DEBUG] [app] Ending request: urn:request:1ccaffd5-a06d-44c6-afe9-f7cd84df3b6d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1ccaffd5-a06d-44c6-afe9-f7cd84df3b6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:14:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:14:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:44,243 [238] [DEBUG] [app] Starting request: urn:request:5ba11763-428f-42e5-b9ee-b90d6f294501 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:44,243 [238] [DEBUG] [app] Ending request: urn:request:5ba11763-428f-42e5-b9ee-b90d6f294501 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ba11763-428f-42e5-b9ee-b90d6f294501', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:14:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:14:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:14:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:44,254 [235] [DEBUG] [app] Ending request: urn:request:e665fb6c-8e4d-4536-8af2-611fea9b6a1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e665fb6c-8e4d-4536-8af2-611fea9b6a1e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:14:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:14:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:14:44,255 [237] [DEBUG] [app] Ending request: urn:request:da923caa-8836-4fc1-b9ac-d2d3788c2801 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:da923caa-8836-4fc1-b9ac-d2d3788c2801', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:14:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:14:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:14:44,402 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:44,794 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:14:45,371 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:14:45,452 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:14:45,615 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:14:45,917 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:14:46,380 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:14:46,488 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:14:46,510 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-web stdout | 2025-02-07 04:14:46,517 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:14:46,522 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:14:46,527 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:14:46,529 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:14:46,532 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:14:47,544 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:14:47,547 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:14:47,551 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:14:48,185 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:14:48,302 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:14:48,617 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:14:48,665 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:14:48,669 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:14:48,674 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:14:48,678 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:14:48,681 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:14:48,683 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:14:48,686 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:14:48,690 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:14:48,692 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:14:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:14:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:50.814677+00:00 (in 2.082007 seconds)
notificationworker stdout | 2025-02-07 04:14:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:58 UTC)" (scheduled at 2025-02-07 04:14:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:14:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:14:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 48, 732994), True, datetime.datetime(2025, 2, 7, 4, 14, 48, 732994), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:14:48,743 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:14:48,743 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:14:48,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:14:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:14:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:14:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:58.732147+00:00 (in 7.916905 seconds)
notificationworker stdout | 2025-02-07 04:14:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:50 UTC)" (scheduled at 2025-02-07 04:14:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:14:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:14:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:14:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:14:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:14:59.937899+00:00 (in 6.999821 seconds)
exportactionlogsworker stdout | 2025-02-07 04:14:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:19:52 UTC)" (scheduled at 2025-02-07 04:14:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:14:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 14, 52, 938343), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:14:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 14, 52, 938343), True, datetime.datetime(2025, 2, 7, 4, 14, 52, 938343), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:14:52,946 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 14, 52, 938343), True, datetime.datetime(2025, 2, 7, 4, 14, 52, 938343), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 14, 52, 938343), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:14:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:14:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:19:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:14:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:14:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:10.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 04:14:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:24 UTC)" (scheduled at 2025-02-07 04:14:54.423055+00:00)
gcworker stdout | 2025-02-07 04:14:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:14:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:14:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:14:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:56.118855+00:00 (in 59.999493 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:14:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:56 UTC)" (scheduled at 2025-02-07 04:14:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:14:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:14:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:14:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:37.715878+00:00 (in 41.001836 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:56 UTC)" (scheduled at 2025-02-07 04:14:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:56,715 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:14:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:14:57,654 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:14:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:14:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:08.732147+00:00 (in 9.999475 seconds)
notificationworker stdout | 2025-02-07 04:14:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:08 UTC)" (scheduled at 2025-02-07 04:14:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:14:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:14:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 14, 58, 732982), True, datetime.datetime(2025, 2, 7, 4, 14, 58, 732982), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:14:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:14:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:14:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:14:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:14:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:14.225378+00:00 (in 14.999518 seconds)
quotatotalworker stdout | 2025-02-07 04:14:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:14 UTC)" (scheduled at 2025-02-07 04:14:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:14:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:14:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:14:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:14:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:14:59,235 [237] [DEBUG] [app] Starting request: urn:request:f954787b-a77e-4580-898a-a84d7f55b068 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:59,235 [236] [DEBUG] [app] Starting request: urn:request:39d1cbbc-8d80-4401-b4de-b2429772a293 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:14:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:14:59,238 [245] [DEBUG] [app] Starting request: urn:request:4cf0af4b-bab3-4114-8b2a-63de4d14879b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:14:59,238 [246] [DEBUG] [app] Starting request: urn:request:ec67e37c-93ba-40e1-878d-2b0875071928 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:14:59,239 [245] [DEBUG] [app] Ending request: urn:request:4cf0af4b-bab3-4114-8b2a-63de4d14879b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4cf0af4b-bab3-4114-8b2a-63de4d14879b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:14:59,239 [246] [DEBUG] [app] Ending request: urn:request:ec67e37c-93ba-40e1-878d-2b0875071928 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec67e37c-93ba-40e1-878d-2b0875071928', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:14:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:14:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:14:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:14:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:14:59,241 [236] [DEBUG] [app] Starting request: urn:request:ee78096d-da94-48dc-bcb7-a14d04141393 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:59,242 [235] [DEBUG] [app] Starting request: urn:request:0cb7fab8-70c7-4296-8eee-baefece1926f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:14:59,242 [236] [DEBUG] [app] Ending request: urn:request:ee78096d-da94-48dc-bcb7-a14d04141393 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee78096d-da94-48dc-bcb7-a14d04141393', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:59,242 [235] [DEBUG] [app] Ending request: urn:request:0cb7fab8-70c7-4296-8eee-baefece1926f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0cb7fab8-70c7-4296-8eee-baefece1926f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:14:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:14:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:14:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:14:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:14:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:14:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:14:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:14:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:14:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:14:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:14:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:14:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:14:59,254 [236] [DEBUG] [app] Ending request: urn:request:39d1cbbc-8d80-4401-b4de-b2429772a293 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39d1cbbc-8d80-4401-b4de-b2429772a293', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:14:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:14:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:14:59,255 [237] [DEBUG] [app] Ending request: urn:request:f954787b-a77e-4580-898a-a84d7f55b068 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f954787b-a77e-4580-898a-a84d7f55b068', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:14:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:14:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:14:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:14:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:14:59,538 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:14:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:14:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:04.936688+00:00 (in 4.998339 seconds)
exportactionlogsworker stdout | 2025-02-07 04:14:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:59 UTC)" (scheduled at 2025-02-07 04:14:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:14:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:14:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:15:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:15:00,659 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:15:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:15:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:23.420886+00:00 (in 20.001287 seconds)
repositorygcworker stdout | 2025-02-07 04:15:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:03 UTC)" (scheduled at 2025-02-07 04:15:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:15:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:15:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 3, 419911), True, datetime.datetime(2025, 2, 7, 4, 15, 3, 419911), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:15:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:15:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:15:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:15:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:15:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:33.919286+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2025-02-07 04:15:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:33 UTC)" (scheduled at 2025-02-07 04:15:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:15:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 15, 3, 920043), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:15:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:15:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:15:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:15:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:15:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:59.937899+00:00 (in 55.000780 seconds)
exportactionlogsworker stdout | 2025-02-07 04:15:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:04 UTC)" (scheduled at 2025-02-07 04:15:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:15:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:15:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 4, 937409), True, datetime.datetime(2025, 2, 7, 4, 15, 4, 937409), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:15:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:15:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:15:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:15:04,979 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:15:06,026 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:15:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:15:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:18.732147+00:00 (in 9.999499 seconds)
notificationworker stdout | 2025-02-07 04:15:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:18 UTC)" (scheduled at 2025-02-07 04:15:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:15:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:15:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 8, 732945), True, datetime.datetime(2025, 2, 7, 4, 15, 8, 732945), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:15:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:15:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:15:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:15:09,946 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:15:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:15:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:24.423055+00:00 (in 13.998346 seconds)
gcworker stdout | 2025-02-07 04:15:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:40 UTC)" (scheduled at 2025-02-07 04:15:10.424250+00:00)
gcworker stdout | 2025-02-07 04:15:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:15:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901410431, None, 1, 0])
gcworker stdout | 2025-02-07 04:15:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:15:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:15:10,604 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:15:11,113 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:15:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:15:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 04:15:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:41 UTC)" (scheduled at 2025-02-07 04:15:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:15:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898111836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:15:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:15:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:15:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:15:12,391 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:15:13,307 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:15:13,331 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:15:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:15:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:29.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 04:15:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:29 UTC)" (scheduled at 2025-02-07 04:15:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:15:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:15:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:15:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:15:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:15:14,235 [235] [DEBUG] [app] Starting request: urn:request:2a8ce355-a589-4655-9fb8-25c719240197 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:14,235 [236] [DEBUG] [app] Starting request: urn:request:f7654565-fbb9-4658-a750-7f0524335d7a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:15:14,238 [239] [DEBUG] [app] Starting request: urn:request:9c5e2127-86ea-49a8-8584-2a340eafe1e0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:14,238 [246] [DEBUG] [app] Starting request: urn:request:e12bdbef-09d9-4219-bc66-3638c0188c73 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:14,238 [239] [DEBUG] [app] Ending request: urn:request:9c5e2127-86ea-49a8-8584-2a340eafe1e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c5e2127-86ea-49a8-8584-2a340eafe1e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:15:14,238 [246] [DEBUG] [app] Ending request: urn:request:e12bdbef-09d9-4219-bc66-3638c0188c73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e12bdbef-09d9-4219-bc66-3638c0188c73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:15:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:15:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:15:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:14,242 [238] [DEBUG] [app] Starting request: urn:request:6ad149ad-2e67-4c38-b701-14d98a09cd97 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:15:14,242 [237] [DEBUG] [app] Starting request: urn:request:623346b7-a3c3-479e-af28-77b78308beca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:15:14,242 [237] [DEBUG] [app] Ending request: urn:request:623346b7-a3c3-479e-af28-77b78308beca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:623346b7-a3c3-479e-af28-77b78308beca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:15:14,242 [238] [DEBUG] [app] Ending request: urn:request:6ad149ad-2e67-4c38-b701-14d98a09cd97 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6ad149ad-2e67-4c38-b701-14d98a09cd97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:15:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:15:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:15:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:15:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:14,254 [236] [DEBUG] [app] Ending request: urn:request:f7654565-fbb9-4658-a750-7f0524335d7a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f7654565-fbb9-4658-a750-7f0524335d7a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:14,254 [235] [DEBUG] [app] Ending request: urn:request:2a8ce355-a589-4655-9fb8-25c719240197 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a8ce355-a589-4655-9fb8-25c719240197', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:15:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:15:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:15:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
autopruneworker stdout | 2025-02-07 04:15:14,437 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:14,830 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:15:15,408 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:15:15,461 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:15:15,640 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:15:15,954 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:15:16,416 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 04:15:16,518 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 04:15:16,524 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-web stdout | 2025-02-07 04:15:16,527 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:15:16,529 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:15:16,533 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:15:16,536 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:15:16,539 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gunicorn-secscan stdout | 2025-02-07 04:15:17,552 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:15:17,556 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:15:17,558 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:15:18,222 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:15:18,338 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:15:18,641 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:15:18,677 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:15:18,680 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:15:18,683 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:15:18,688 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:15:18,691 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:15:18,695 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:15:18,698 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:15:18,700 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:15:18,703 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:15:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:15:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:28.732147+00:00 (in 9.999534 seconds)
notificationworker stdout | 2025-02-07 04:15:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:28 UTC)" (scheduled at 2025-02-07 04:15:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:15:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:15:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 18, 732908), True, datetime.datetime(2025, 2, 7, 4, 15, 18, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:15:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:15:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:15:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:15:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:15:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:03.419110+00:00 (in 39.997714 seconds)
repositorygcworker stdout | 2025-02-07 04:15:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:23 UTC)" (scheduled at 2025-02-07 04:15:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:15:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:15:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:15:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:15:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:40.424250+00:00 (in 16.000744 seconds)
gcworker stdout | 2025-02-07 04:15:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:54 UTC)" (scheduled at 2025-02-07 04:15:24.423055+00:00)
gcworker stdout | 2025-02-07 04:15:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:15:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:15:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:15:27,690 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:15:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:15:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:38.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:15:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:38 UTC)" (scheduled at 2025-02-07 04:15:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:15:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:15:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 28, 732864), True, datetime.datetime(2025, 2, 7, 4, 15, 28, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:15:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:15:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:15:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:15:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:15:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:44.225378+00:00 (in 14.999568 seconds)
quotatotalworker stdout | 2025-02-07 04:15:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:44 UTC)" (scheduled at 2025-02-07 04:15:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:15:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:15:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:15:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:15:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:15:29,235 [235] [DEBUG] [app] Starting request: urn:request:bcd7a11b-dd9b-4b44-86a0-bceb6320ec30 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:29,235 [238] [DEBUG] [app] Starting request: urn:request:76fdb896-c0b4-4d26-bead-1057dc7420ca (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:15:29,238 [246] [DEBUG] [app] Starting request: urn:request:a3b640aa-6eb6-4063-91b7-c85e9718e2cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:29,238 [239] [DEBUG] [app] Starting request: urn:request:050fbe65-307a-42cd-8906-68d72c620ba6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:29,239 [246] [DEBUG] [app] Ending request: urn:request:a3b640aa-6eb6-4063-91b7-c85e9718e2cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3b640aa-6eb6-4063-91b7-c85e9718e2cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:15:29,239 [239] [DEBUG] [app] Ending request: urn:request:050fbe65-307a-42cd-8906-68d72c620ba6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:050fbe65-307a-42cd-8906-68d72c620ba6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:15:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:15:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:15:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:29,241 [237] [DEBUG] [app] Starting request: urn:request:96bc5f0b-5d79-485c-8e24-f2a89645b453 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:15:29,242 [237] [DEBUG] [app] Ending request: urn:request:96bc5f0b-5d79-485c-8e24-f2a89645b453 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:96bc5f0b-5d79-485c-8e24-f2a89645b453', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:15:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:15:29,242 [235] [DEBUG] [app] Starting request: urn:request:e7a042b4-4fce-4a00-98bf-4b5c15a97fd1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:15:29,242 [235] [DEBUG] [app] Ending request: urn:request:e7a042b4-4fce-4a00-98bf-4b5c15a97fd1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7a042b4-4fce-4a00-98bf-4b5c15a97fd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:15:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:15:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:15:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:29,254 [235] [DEBUG] [app] Ending request: urn:request:bcd7a11b-dd9b-4b44-86a0-bceb6320ec30 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bcd7a11b-dd9b-4b44-86a0-bceb6320ec30', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:29,254 [238] [DEBUG] [app] Ending request: urn:request:76fdb896-c0b4-4d26-bead-1057dc7420ca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76fdb896-c0b4-4d26-bead-1057dc7420ca', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:15:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:15:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:15:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:15:29,575 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:15:30,696 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:15:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:15:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:32.625945+00:00 (in 1.004315 seconds)
namespacegcworker stdout | 2025-02-07 04:15:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:31 UTC)" (scheduled at 2025-02-07 04:15:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:15:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:15:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 31, 621906), True, datetime.datetime(2025, 2, 7, 4, 15, 31, 621906), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:15:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:15:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:15:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:15:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:15:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:31.621131+00:00 (in 58.994724 seconds)
namespacegcworker stdout | 2025-02-07 04:15:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:32 UTC)" (scheduled at 2025-02-07 04:15:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:15:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:15:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:15:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:15:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:03.919286+00:00 (in 29.999556 seconds)
buildlogsarchiver stdout | 2025-02-07 04:15:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:03 UTC)" (scheduled at 2025-02-07 04:15:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:15:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 15, 33, 919982), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:15:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:15:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:15:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:15:35,015 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:15:36,057 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:56.713547+00:00 (in 18.997205 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:37 UTC)" (scheduled at 2025-02-07 04:15:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:15:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:15:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:48.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:15:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:48 UTC)" (scheduled at 2025-02-07 04:15:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:15:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:15:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 38, 732894), True, datetime.datetime(2025, 2, 7, 4, 15, 38, 732894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:15:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:15:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:15:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:15:39,971 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:15:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:15:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:54.423055+00:00 (in 13.998340 seconds)
gcworker stdout | 2025-02-07 04:15:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:10 UTC)" (scheduled at 2025-02-07 04:15:40.424250+00:00)
gcworker stdout | 2025-02-07 04:15:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:15:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901440431, None, 1, 0])
gcworker stdout | 2025-02-07 04:15:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:15:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:15:40,629 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:15:41,146 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:15:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:15:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:11.831521+00:00 (in 29.999564 seconds)
autopruneworker stdout | 2025-02-07 04:15:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:11 UTC)" (scheduled at 2025-02-07 04:15:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:15:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898141836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:15:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:15:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:15:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:15:42,401 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:15:43,331 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:15:43,367 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:15:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:15:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:59.225378+00:00 (in 14.999436 seconds)
quotatotalworker stdout | 2025-02-07 04:15:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:59 UTC)" (scheduled at 2025-02-07 04:15:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:15:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:15:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:15:44,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:15:44,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:15:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:15:44,236 [235] [DEBUG] [app] Starting request: urn:request:2d09534e-1120-457b-9885-05f33f0a8500 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:44,236 [238] [DEBUG] [app] Starting request: urn:request:426647fc-fc63-4597-bff6-f61c124761e0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:15:44,240 [245] [DEBUG] [app] Starting request: urn:request:6140efd4-530c-4e16-802b-dc9588b61cec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:44,240 [246] [DEBUG] [app] Starting request: urn:request:c06657a5-e44c-40ac-97c7-2afd6152cdf6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:44,240 [245] [DEBUG] [app] Ending request: urn:request:6140efd4-530c-4e16-802b-dc9588b61cec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6140efd4-530c-4e16-802b-dc9588b61cec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:15:44,240 [246] [DEBUG] [app] Ending request: urn:request:c06657a5-e44c-40ac-97c7-2afd6152cdf6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c06657a5-e44c-40ac-97c7-2afd6152cdf6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:15:44,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:15:44,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:44,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:44,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:44,244 [237] [DEBUG] [app] Starting request: urn:request:bc955ef7-8c0d-4b97-9ed8-a7d6b0bafb50 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:15:44,244 [237] [DEBUG] [app] Ending request: urn:request:bc955ef7-8c0d-4b97-9ed8-a7d6b0bafb50 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc955ef7-8c0d-4b97-9ed8-a7d6b0bafb50', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:15:44,245 [235] [DEBUG] [app] Starting request: urn:request:000235de-c3f9-426d-903c-929289bf3555 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:15:44,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:44,245 [235] [DEBUG] [app] Ending request: urn:request:000235de-c3f9-426d-903c-929289bf3555 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:000235de-c3f9-426d-903c-929289bf3555', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:15:44,245 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:44,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:44,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:44,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:44,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:15:44,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:15:44,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:44,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:44,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:44,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:44,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:44,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:44,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:44,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:44,257 [235] [DEBUG] [app] Ending request: urn:request:2d09534e-1120-457b-9885-05f33f0a8500 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d09534e-1120-457b-9885-05f33f0a8500', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:44,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:15:44,257 [238] [DEBUG] [app] Ending request: urn:request:426647fc-fc63-4597-bff6-f61c124761e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:426647fc-fc63-4597-bff6-f61c124761e0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:44,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:15:44,257 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.022)
gunicorn-web stdout | 2025-02-07 04:15:44,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.022)
autopruneworker stdout | 2025-02-07 04:15:44,466 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:44,860 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:15:45,435 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:15:45,497 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:15:45,659 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:15:45,984 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:15:46,453 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-web stdout | 2025-02-07 04:15:46,535 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:15:46,537 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:15:46,540 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:15:46,543 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:15:46,545 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestbackfillworker stdout | 2025-02-07 04:15:46,553 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 04:15:46,558 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-secscan stdout | 2025-02-07 04:15:47,561 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:15:47,563 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:15:47,566 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:15:48,259 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:15:48,351 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:15:48,678 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:15:48,686 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:15:48,690 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:15:48,692 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:15:48,696 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:15:48,699 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:15:48,701 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:15:48,706 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:15:48,709 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:15:48,711 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:15:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:15:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:50.814677+00:00 (in 2.082096 seconds)
notificationworker stdout | 2025-02-07 04:15:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:58 UTC)" (scheduled at 2025-02-07 04:15:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:15:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:15:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 48, 732875), True, datetime.datetime(2025, 2, 7, 4, 15, 48, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:15:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:15:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:15:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:15:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:15:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:15:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:15:58.732147+00:00 (in 7.916988 seconds)
notificationworker stdout | 2025-02-07 04:15:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:50 UTC)" (scheduled at 2025-02-07 04:15:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:15:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:15:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:15:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:15:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:10.424250+00:00 (in 16.000731 seconds)
gcworker stdout | 2025-02-07 04:15:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:24 UTC)" (scheduled at 2025-02-07 04:15:54.423055+00:00)
gcworker stdout | 2025-02-07 04:15:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:15:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:15:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:15:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:56.118855+00:00 (in 59.999515 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:15:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:56 UTC)" (scheduled at 2025-02-07 04:15:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:15:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:15:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:15:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:37.715878+00:00 (in 41.001871 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:56 UTC)" (scheduled at 2025-02-07 04:15:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:15:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:15:57,726 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:15:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:15:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:08.732147+00:00 (in 9.999511 seconds)
notificationworker stdout | 2025-02-07 04:15:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:08 UTC)" (scheduled at 2025-02-07 04:15:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:15:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:15:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 15, 58, 732912), True, datetime.datetime(2025, 2, 7, 4, 15, 58, 732912), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:15:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:15:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:15:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:15:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:15:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:14.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:15:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:14 UTC)" (scheduled at 2025-02-07 04:15:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:15:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:15:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:15:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:15:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:15:59,235 [235] [DEBUG] [app] Starting request: urn:request:25b354c1-a8c0-4bce-baa6-1d0e6b5e970e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:59,235 [237] [DEBUG] [app] Starting request: urn:request:78aee886-15fa-4452-b7b7-cf8214639bba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:15:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:15:59,238 [246] [DEBUG] [app] Starting request: urn:request:92ef41e2-233a-495c-87f0-f6682f9cdde0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:59,238 [239] [DEBUG] [app] Starting request: urn:request:89d8ee31-e527-4cc2-9e2f-40a8d6441fb2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:15:59,239 [239] [DEBUG] [app] Ending request: urn:request:89d8ee31-e527-4cc2-9e2f-40a8d6441fb2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:89d8ee31-e527-4cc2-9e2f-40a8d6441fb2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:15:59,239 [246] [DEBUG] [app] Ending request: urn:request:92ef41e2-233a-495c-87f0-f6682f9cdde0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:92ef41e2-233a-495c-87f0-f6682f9cdde0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:15:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:15:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:15:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:15:59,241 [237] [DEBUG] [app] Starting request: urn:request:3c5f3bf9-b507-4011-b824-a01061253bd8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:15:59,242 [237] [DEBUG] [app] Ending request: urn:request:3c5f3bf9-b507-4011-b824-a01061253bd8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c5f3bf9-b507-4011-b824-a01061253bd8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:15:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:59,242 [238] [DEBUG] [app] Starting request: urn:request:d05eb8f9-c858-4c76-8f67-e61186ea0449 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:15:59,242 [238] [DEBUG] [app] Ending request: urn:request:d05eb8f9-c858-4c76-8f67-e61186ea0449 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d05eb8f9-c858-4c76-8f67-e61186ea0449', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:15:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:15:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:15:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:15:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:15:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:15:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:15:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:15:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:15:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:15:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:15:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:15:59,253 [237] [DEBUG] [app] Ending request: urn:request:78aee886-15fa-4452-b7b7-cf8214639bba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78aee886-15fa-4452-b7b7-cf8214639bba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:59,253 [235] [DEBUG] [app] Ending request: urn:request:25b354c1-a8c0-4bce-baa6-1d0e6b5e970e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:25b354c1-a8c0-4bce-baa6-1d0e6b5e970e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:15:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:15:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:15:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:15:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:15:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:15:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:15:59,591 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:15:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:15:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:04.936688+00:00 (in 4.998333 seconds)
exportactionlogsworker stdout | 2025-02-07 04:15:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:59 UTC)" (scheduled at 2025-02-07 04:15:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:15:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:15:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:16:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:16:00,733 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:16:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:16:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:23.420886+00:00 (in 20.001325 seconds)
repositorygcworker stdout | 2025-02-07 04:16:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:03 UTC)" (scheduled at 2025-02-07 04:16:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:16:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:16:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 3, 419854), True, datetime.datetime(2025, 2, 7, 4, 16, 3, 419854), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:16:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:16:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:16:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:16:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:16:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:33.919286+00:00 (in 29.999574 seconds)
buildlogsarchiver stdout | 2025-02-07 04:16:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:33 UTC)" (scheduled at 2025-02-07 04:16:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:16:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 16, 3, 919988), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:16:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:16:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:16:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:16:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:16:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:59.937899+00:00 (in 55.000780 seconds)
exportactionlogsworker stdout | 2025-02-07 04:16:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:04 UTC)" (scheduled at 2025-02-07 04:16:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:16:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:16:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 4, 937459), True, datetime.datetime(2025, 2, 7, 4, 16, 4, 937459), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:16:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:16:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:16:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:16:05,052 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:16:06,093 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:16:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:16:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:18.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 04:16:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:18 UTC)" (scheduled at 2025-02-07 04:16:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:16:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:16:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 8, 732874), True, datetime.datetime(2025, 2, 7, 4, 16, 8, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:16:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:16:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:16:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:16:09,999 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:16:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:16:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:24.423055+00:00 (in 13.998328 seconds)
gcworker stdout | 2025-02-07 04:16:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:40 UTC)" (scheduled at 2025-02-07 04:16:10.424250+00:00)
gcworker stdout | 2025-02-07 04:16:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:16:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901470431, None, 1, 0])
gcworker stdout | 2025-02-07 04:16:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:16:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:16:10,658 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:16:11,178 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:16:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:16:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:41.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 04:16:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:41 UTC)" (scheduled at 2025-02-07 04:16:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:16:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898171836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:16:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:16:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:16:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:16:12,426 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:16:13,368 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:16:13,386 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:16:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:16:14,226 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:29.225378+00:00 (in 14.999365 seconds)
quotatotalworker stdout | 2025-02-07 04:16:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:29 UTC)" (scheduled at 2025-02-07 04:16:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:16:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:16:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:16:14,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:16:14,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:16:14,236 [235] [DEBUG] [app] Starting request: urn:request:a771a1f3-ce15-4289-8003-dd478514ab87 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:14,236 [238] [DEBUG] [app] Starting request: urn:request:50288f4f-d6f6-4342-8270-0c71777f3369 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:16:14,240 [239] [DEBUG] [app] Starting request: urn:request:98cedc85-fe39-49b0-8e6a-9dd7d6667337 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:14,240 [246] [DEBUG] [app] Starting request: urn:request:48604243-0c67-4024-bfd6-b373cee98de9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:14,241 [239] [DEBUG] [app] Ending request: urn:request:98cedc85-fe39-49b0-8e6a-9dd7d6667337 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:98cedc85-fe39-49b0-8e6a-9dd7d6667337', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:16:14,241 [246] [DEBUG] [app] Ending request: urn:request:48604243-0c67-4024-bfd6-b373cee98de9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48604243-0c67-4024-bfd6-b373cee98de9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:16:14,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:16:14,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:14,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:14,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:14,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:14,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:14,245 [237] [DEBUG] [app] Starting request: urn:request:dc8bebbf-0dd3-4e05-a78a-9d2fc1bfb416 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:14,245 [235] [DEBUG] [app] Starting request: urn:request:bd410af5-1b0d-43d4-8c5d-8408800d86cd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:14,245 [235] [DEBUG] [app] Ending request: urn:request:bd410af5-1b0d-43d4-8c5d-8408800d86cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bd410af5-1b0d-43d4-8c5d-8408800d86cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:16:14,245 [237] [DEBUG] [app] Ending request: urn:request:dc8bebbf-0dd3-4e05-a78a-9d2fc1bfb416 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dc8bebbf-0dd3-4e05-a78a-9d2fc1bfb416', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:16:14,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:16:14,246 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:14,246 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:14,246 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:14,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:14,247 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:14,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:16:14,247 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:16:14,251 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:14,251 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:14,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:14,256 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:14,256 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:14,257 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:14,257 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:14,258 [238] [DEBUG] [app] Ending request: urn:request:50288f4f-d6f6-4342-8270-0c71777f3369 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:50288f4f-d6f6-4342-8270-0c71777f3369', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:14,258 [235] [DEBUG] [app] Ending request: urn:request:a771a1f3-ce15-4289-8003-dd478514ab87 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a771a1f3-ce15-4289-8003-dd478514ab87', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:14,259 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:16:14,259 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gunicorn-web stdout | 2025-02-07 04:16:14,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:16:14,259 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:16:14,502 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:14,897 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:16:15,472 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:16:15,522 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:16:15,695 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:16:16,020 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:16:16,482 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-web stdout | 2025-02-07 04:16:16,543 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:16:16,547 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:16:16,550 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:16:16,553 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:16:16,557 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestbackfillworker stdout | 2025-02-07 04:16:16,570 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 04:16:16,580 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-secscan stdout | 2025-02-07 04:16:17,568 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:16:17,571 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:16:17,574 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:16:18,274 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:16:18,387 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:16:18,696 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:16:18,699 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:16:18,702 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:16:18,706 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:16:18,709 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:16:18,711 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
blobuploadcleanupworker stdout | 2025-02-07 04:16:18,714 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gunicorn-registry stdout | 2025-02-07 04:16:18,717 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:16:18,720 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:16:18,722 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:16:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:16:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:28.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 04:16:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:28 UTC)" (scheduled at 2025-02-07 04:16:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:16:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:16:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 18, 732836), True, datetime.datetime(2025, 2, 7, 4, 16, 18, 732836), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:16:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:16:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:16:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:16:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:16:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:03.419110+00:00 (in 39.997745 seconds)
repositorygcworker stdout | 2025-02-07 04:16:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:23 UTC)" (scheduled at 2025-02-07 04:16:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:16:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:16:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:16:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:16:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:40.424250+00:00 (in 16.000749 seconds)
gcworker stdout | 2025-02-07 04:16:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:54 UTC)" (scheduled at 2025-02-07 04:16:24.423055+00:00)
gcworker stdout | 2025-02-07 04:16:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:16:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:16:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:16:27,753 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:16:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:16:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:38.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 04:16:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:38 UTC)" (scheduled at 2025-02-07 04:16:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:16:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:16:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 28, 732857), True, datetime.datetime(2025, 2, 7, 4, 16, 28, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:16:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:16:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:16:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:16:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:16:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:44.225378+00:00 (in 14.999477 seconds)
quotatotalworker stdout | 2025-02-07 04:16:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:44 UTC)" (scheduled at 2025-02-07 04:16:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:16:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:16:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:16:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:16:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:16:29,235 [238] [DEBUG] [app] Starting request: urn:request:b5c41122-8a8a-4324-91a1-349c4685051b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:29,235 [235] [DEBUG] [app] Starting request: urn:request:5c8983ff-cdd8-4823-a3cb-e333d46cd140 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:16:29,239 [239] [DEBUG] [app] Starting request: urn:request:c5eaeb2b-4852-43a9-ae0a-e56064f2b91b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:29,239 [245] [DEBUG] [app] Starting request: urn:request:bd9bc3ec-8a63-4a8a-8d02-acb6d628fba5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:29,239 [239] [DEBUG] [app] Ending request: urn:request:c5eaeb2b-4852-43a9-ae0a-e56064f2b91b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5eaeb2b-4852-43a9-ae0a-e56064f2b91b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:16:29,239 [245] [DEBUG] [app] Ending request: urn:request:bd9bc3ec-8a63-4a8a-8d02-acb6d628fba5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd9bc3ec-8a63-4a8a-8d02-acb6d628fba5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:16:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:16:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:16:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:29,242 [237] [DEBUG] [app] Starting request: urn:request:7a8e7824-d1b4-4b60-9d09-772c047b5bc0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:29,242 [237] [DEBUG] [app] Ending request: urn:request:7a8e7824-d1b4-4b60-9d09-772c047b5bc0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a8e7824-d1b4-4b60-9d09-772c047b5bc0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:16:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:29,243 [236] [DEBUG] [app] Starting request: urn:request:622567fa-915a-4110-b206-23c522ddaaeb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:29,243 [236] [DEBUG] [app] Ending request: urn:request:622567fa-915a-4110-b206-23c522ddaaeb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:622567fa-915a-4110-b206-23c522ddaaeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:16:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:16:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:16:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:16:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:29,254 [238] [DEBUG] [app] Ending request: urn:request:b5c41122-8a8a-4324-91a1-349c4685051b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5c41122-8a8a-4324-91a1-349c4685051b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:16:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:16:29,254 [235] [DEBUG] [app] Ending request: urn:request:5c8983ff-cdd8-4823-a3cb-e333d46cd140 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5c8983ff-cdd8-4823-a3cb-e333d46cd140', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:16:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
repositoryactioncounter stdout | 2025-02-07 04:16:29,627 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:16:30,750 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:16:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:16:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:32.625945+00:00 (in 1.004350 seconds)
namespacegcworker stdout | 2025-02-07 04:16:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:31 UTC)" (scheduled at 2025-02-07 04:16:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:16:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:16:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 31, 621871), True, datetime.datetime(2025, 2, 7, 4, 16, 31, 621871), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:16:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:16:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:16:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:16:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:16:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:13.624301+00:00 (in 40.997872 seconds)
namespacegcworker stdout | 2025-02-07 04:16:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:32 UTC)" (scheduled at 2025-02-07 04:16:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:16:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:16:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:16:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:16:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:03.919286+00:00 (in 29.999554 seconds)
buildlogsarchiver stdout | 2025-02-07 04:16:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:03 UTC)" (scheduled at 2025-02-07 04:16:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:16:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 16, 33, 919985), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:16:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:16:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:16:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:16:35,089 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:16:36,129 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:56.713547+00:00 (in 18.997167 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:37 UTC)" (scheduled at 2025-02-07 04:16:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:16:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:16:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:48.732147+00:00 (in 9.999530 seconds)
notificationworker stdout | 2025-02-07 04:16:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:48 UTC)" (scheduled at 2025-02-07 04:16:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:16:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:16:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 38, 732889), True, datetime.datetime(2025, 2, 7, 4, 16, 38, 732889), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:16:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:16:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:16:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:16:40,010 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:16:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:16:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:54.423055+00:00 (in 13.998371 seconds)
gcworker stdout | 2025-02-07 04:16:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:10 UTC)" (scheduled at 2025-02-07 04:16:40.424250+00:00)
gcworker stdout | 2025-02-07 04:16:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:16:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901500431, None, 1, 0])
gcworker stdout | 2025-02-07 04:16:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:16:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:16:40,695 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:16:41,186 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:16:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:16:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:11.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 04:16:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:11 UTC)" (scheduled at 2025-02-07 04:16:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:16:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898201836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:16:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:16:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:16:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:16:42,462 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:16:43,388 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:16:43,421 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:16:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:16:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:59.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 04:16:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:59 UTC)" (scheduled at 2025-02-07 04:16:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:16:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:16:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:16:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:16:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:16:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:16:44,236 [235] [DEBUG] [app] Starting request: urn:request:7771f347-666a-4eb7-ad13-e3c64021f366 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:44,236 [238] [DEBUG] [app] Starting request: urn:request:7c4d0af4-0934-48de-be40-9e1b9f91058d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:44,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:16:44,239 [239] [DEBUG] [app] Starting request: urn:request:cd0807bb-aab7-4297-9f33-512829bc82d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:44,239 [246] [DEBUG] [app] Starting request: urn:request:74bd9a3c-4d51-48a4-85a9-c22a8bcdf433 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:44,240 [239] [DEBUG] [app] Ending request: urn:request:cd0807bb-aab7-4297-9f33-512829bc82d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cd0807bb-aab7-4297-9f33-512829bc82d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:16:44,240 [246] [DEBUG] [app] Ending request: urn:request:74bd9a3c-4d51-48a4-85a9-c22a8bcdf433 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:74bd9a3c-4d51-48a4-85a9-c22a8bcdf433', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:16:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:16:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:44,243 [238] [DEBUG] [app] Starting request: urn:request:2d5a3c8a-4ac4-4475-9937-d0ecff64d965 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:44,243 [238] [DEBUG] [app] Ending request: urn:request:2d5a3c8a-4ac4-4475-9937-d0ecff64d965 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2d5a3c8a-4ac4-4475-9937-d0ecff64d965', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:16:44,243 [237] [DEBUG] [app] Starting request: urn:request:f320809e-4443-4316-b505-fa7601115bc6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:44,243 [237] [DEBUG] [app] Ending request: urn:request:f320809e-4443-4316-b505-fa7601115bc6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f320809e-4443-4316-b505-fa7601115bc6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:16:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:16:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:16:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:44,254 [238] [DEBUG] [app] Ending request: urn:request:7c4d0af4-0934-48de-be40-9e1b9f91058d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c4d0af4-0934-48de-be40-9e1b9f91058d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:16:44,255 [235] [DEBUG] [app] Ending request: urn:request:7771f347-666a-4eb7-ad13-e3c64021f366 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7771f347-666a-4eb7-ad13-e3c64021f366', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:16:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:16:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:16:44,521 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:44,919 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:16:45,492 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:16:45,558 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:16:45,715 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:16:46,029 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:16:46,518 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-web stdout | 2025-02-07 04:16:46,551 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:16:46,554 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:16:46,557 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:16:46,560 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:16:46,564 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
securityscanningnotificationworker stdout | 2025-02-07 04:16:46,594 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:16:46,605 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:16:47,577 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:16:47,579 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:16:47,583 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:16:48,310 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:16:48,417 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:16:48,706 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:16:48,709 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:16:48,712 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:16:48,715 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:16:48,718 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:16:48,721 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:16:48,723 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:16:48,729 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
notificationworker stdout | 2025-02-07 04:16:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:16:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:50.814677+00:00 (in 2.082161 seconds)
notificationworker stdout | 2025-02-07 04:16:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:58 UTC)" (scheduled at 2025-02-07 04:16:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:16:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
gunicorn-registry stdout | 2025-02-07 04:16:48,733 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
notificationworker stdout | 2025-02-07 04:16:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 48, 732768), True, datetime.datetime(2025, 2, 7, 4, 16, 48, 732768), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:16:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:16:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:16:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:16:58 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:16:48,750 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:16:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:16:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:16:58.732147+00:00 (in 7.917006 seconds)
notificationworker stdout | 2025-02-07 04:16:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:50 UTC)" (scheduled at 2025-02-07 04:16:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:16:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:16:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:16:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:16:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:10.424250+00:00 (in 16.000721 seconds)
gcworker stdout | 2025-02-07 04:16:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:24 UTC)" (scheduled at 2025-02-07 04:16:54.423055+00:00)
gcworker stdout | 2025-02-07 04:16:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:16:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:16:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:16:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:56.118855+00:00 (in 59.999499 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:16:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:56 UTC)" (scheduled at 2025-02-07 04:16:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:16:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:16:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:16:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:37.715878+00:00 (in 41.001854 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:56 UTC)" (scheduled at 2025-02-07 04:16:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:56,715 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:16:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:16:57,789 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:16:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:16:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:08.732147+00:00 (in 9.999528 seconds)
notificationworker stdout | 2025-02-07 04:16:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:08 UTC)" (scheduled at 2025-02-07 04:16:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:16:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:16:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 16, 58, 732904), True, datetime.datetime(2025, 2, 7, 4, 16, 58, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:16:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:16:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:16:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:16:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:16:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:14.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 04:16:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:14 UTC)" (scheduled at 2025-02-07 04:16:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:16:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:16:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:16:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:16:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:16:59,236 [237] [DEBUG] [app] Starting request: urn:request:b2cbb789-0b9b-4c69-a648-aff35955a2c0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:59,236 [238] [DEBUG] [app] Starting request: urn:request:ae60f2b4-942c-433d-baee-1c62da442ab8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:16:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:16:59,239 [239] [DEBUG] [app] Starting request: urn:request:c1e7a364-ae8d-47b0-b6b2-e4ad01d52361 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:59,239 [246] [DEBUG] [app] Starting request: urn:request:da3e5eed-6d38-45d7-9b91-5e3d48405d52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:16:59,239 [239] [DEBUG] [app] Ending request: urn:request:c1e7a364-ae8d-47b0-b6b2-e4ad01d52361 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c1e7a364-ae8d-47b0-b6b2-e4ad01d52361', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:16:59,239 [246] [DEBUG] [app] Ending request: urn:request:da3e5eed-6d38-45d7-9b91-5e3d48405d52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da3e5eed-6d38-45d7-9b91-5e3d48405d52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:16:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:16:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:16:59,242 [235] [DEBUG] [app] Starting request: urn:request:6a5c12c0-47ce-4a58-a2cf-25cd5844ff8a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:59,243 [238] [DEBUG] [app] Starting request: urn:request:70a959cf-d838-4aa5-b2b8-af49560c814b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:16:59,243 [235] [DEBUG] [app] Ending request: urn:request:6a5c12c0-47ce-4a58-a2cf-25cd5844ff8a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a5c12c0-47ce-4a58-a2cf-25cd5844ff8a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:16:59,243 [238] [DEBUG] [app] Ending request: urn:request:70a959cf-d838-4aa5-b2b8-af49560c814b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70a959cf-d838-4aa5-b2b8-af49560c814b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:16:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:16:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:16:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:16:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:16:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:16:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:16:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:16:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:16:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:16:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:16:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:16:59,255 [237] [DEBUG] [app] Ending request: urn:request:b2cbb789-0b9b-4c69-a648-aff35955a2c0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2cbb789-0b9b-4c69-a648-aff35955a2c0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:59,255 [238] [DEBUG] [app] Ending request: urn:request:ae60f2b4-942c-433d-baee-1c62da442ab8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ae60f2b4-942c-433d-baee-1c62da442ab8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:16:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:16:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:16:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:16:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:16:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:16:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
repositoryactioncounter stdout | 2025-02-07 04:16:59,659 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:16:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:16:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:04.936688+00:00 (in 4.998332 seconds)
exportactionlogsworker stdout | 2025-02-07 04:16:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:59 UTC)" (scheduled at 2025-02-07 04:16:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:16:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:16:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:17:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:17:00,759 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:17:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:17:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:23.420886+00:00 (in 20.001298 seconds)
repositorygcworker stdout | 2025-02-07 04:17:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:03 UTC)" (scheduled at 2025-02-07 04:17:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:17:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:17:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 3, 419870), True, datetime.datetime(2025, 2, 7, 4, 17, 3, 419870), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:17:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:17:03,428 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:17:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:17:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:17:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:33.919286+00:00 (in 29.999525 seconds)
buildlogsarchiver stdout | 2025-02-07 04:17:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:33 UTC)" (scheduled at 2025-02-07 04:17:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:17:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 17, 3, 920028), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:17:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:17:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:17:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:17:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:17:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:59.937899+00:00 (in 55.000789 seconds)
exportactionlogsworker stdout | 2025-02-07 04:17:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:04 UTC)" (scheduled at 2025-02-07 04:17:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:17:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:17:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 4, 937399), True, datetime.datetime(2025, 2, 7, 4, 17, 4, 937399), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:17:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:17:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:17:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:17:05,125 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:17:06,165 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:17:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:17:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:18.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 04:17:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:18 UTC)" (scheduled at 2025-02-07 04:17:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:17:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:17:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 8, 732891), True, datetime.datetime(2025, 2, 7, 4, 17, 8, 732891), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:17:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:17:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:17:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:17:10,046 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:17:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:17:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:24.423055+00:00 (in 13.998333 seconds)
gcworker stdout | 2025-02-07 04:17:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:40 UTC)" (scheduled at 2025-02-07 04:17:10.424250+00:00)
gcworker stdout | 2025-02-07 04:17:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:17:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901530431, None, 1, 0])
gcworker stdout | 2025-02-07 04:17:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:17:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:17:10,730 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:17:11,223 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:17:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:17:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:41.831521+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2025-02-07 04:17:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:41 UTC)" (scheduled at 2025-02-07 04:17:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:17:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898231836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:17:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:17:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:17:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:17:12,498 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:17:13,401 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:17:13,456 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
namespacegcworker stdout | 2025-02-07 04:17:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:17:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:31.621131+00:00 (in 17.996342 seconds)
namespacegcworker stdout | 2025-02-07 04:17:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:22:13 UTC)" (scheduled at 2025-02-07 04:17:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:17:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 17, 13, 625069), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:17:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 17, 13, 625069), True, datetime.datetime(2025, 2, 7, 4, 17, 13, 625069), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:17:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 17, 13, 625069), True, datetime.datetime(2025, 2, 7, 4, 17, 13, 625069), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 17, 13, 625069), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:17:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:17:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:22:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:17:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:17:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:29.225378+00:00 (in 14.999480 seconds)
quotatotalworker stdout | 2025-02-07 04:17:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:29 UTC)" (scheduled at 2025-02-07 04:17:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:17:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:17:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:17:14,234 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:17:14,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:17:14,236 [237] [DEBUG] [app] Starting request: urn:request:9d379796-3ff6-4e08-8c5d-b2953928b362 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:14,236 [238] [DEBUG] [app] Starting request: urn:request:e340a8f1-07ec-49d9-a9d0-a0aa74d5be63 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:17:14,239 [239] [DEBUG] [app] Starting request: urn:request:479dc086-98c2-400d-a325-3aef2af00e90 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:14,240 [246] [DEBUG] [app] Starting request: urn:request:347e8c29-1ae5-4bda-99db-5368efd649b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:14,240 [239] [DEBUG] [app] Ending request: urn:request:479dc086-98c2-400d-a325-3aef2af00e90 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:479dc086-98c2-400d-a325-3aef2af00e90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:17:14,240 [246] [DEBUG] [app] Ending request: urn:request:347e8c29-1ae5-4bda-99db-5368efd649b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:347e8c29-1ae5-4bda-99db-5368efd649b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:17:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:17:14,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:17:14,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:14,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:17:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:14,243 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:14,244 [235] [DEBUG] [app] Starting request: urn:request:7031f72b-3b7c-480e-aaa5-7346f30669cb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:14,245 [235] [DEBUG] [app] Ending request: urn:request:7031f72b-3b7c-480e-aaa5-7346f30669cb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7031f72b-3b7c-480e-aaa5-7346f30669cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:17:14,245 [238] [DEBUG] [app] Starting request: urn:request:3db257d7-f79c-45e6-b4cf-1353d16a15ea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:14,245 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:17:14,245 [238] [DEBUG] [app] Ending request: urn:request:3db257d7-f79c-45e6-b4cf-1353d16a15ea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3db257d7-f79c-45e6-b4cf-1353d16a15ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:17:14,246 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:17:14,246 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:17:14,246 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:14,247 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:14,247 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:14,247 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:17:14,247 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:17:14,251 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:14,251 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:14,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:14,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:14,256 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:14,256 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:14,257 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:14,257 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:14,259 [238] [DEBUG] [app] Ending request: urn:request:e340a8f1-07ec-49d9-a9d0-a0aa74d5be63 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e340a8f1-07ec-49d9-a9d0-a0aa74d5be63', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:14,259 [237] [DEBUG] [app] Ending request: urn:request:9d379796-3ff6-4e08-8c5d-b2953928b362 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d379796-3ff6-4e08-8c5d-b2953928b362', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:14,259 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:14,259 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:14,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.025)
gunicorn-web stdout | 2025-02-07 04:17:14,260 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:17:14,558 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:14,954 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:17:15,529 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:17:15,574 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:17:15,730 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:17:16,065 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:17:16,555 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-web stdout | 2025-02-07 04:17:16,559 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:17:16,563 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:17:16,566 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:17:16,569 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:17:16,572 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
securityscanningnotificationworker stdout | 2025-02-07 04:17:16,629 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:17:16,642 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:17:17,585 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:17:17,588 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:17:17,591 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:17:18,332 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:17:18,454 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:17:18,714 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:17:18,717 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:17:18,722 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:17:18,725 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:17:18,730 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
notificationworker stdout | 2025-02-07 04:17:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:17:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:28.732147+00:00 (in 9.999588 seconds)
notificationworker stdout | 2025-02-07 04:17:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:28 UTC)" (scheduled at 2025-02-07 04:17:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:17:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
gunicorn-registry stdout | 2025-02-07 04:17:18,733 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 04:17:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 18, 732823), True, datetime.datetime(2025, 2, 7, 4, 17, 18, 732823), 0, 'notification/%', 50, 1, 0])
gunicorn-registry stdout | 2025-02-07 04:17:18,736 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:17:18,739 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
notificationworker stdout | 2025-02-07 04:17:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:17:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:17:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:17:18,742 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
blobuploadcleanupworker stdout | 2025-02-07 04:17:18,774 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:17:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:17:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:03.419110+00:00 (in 39.997742 seconds)
repositorygcworker stdout | 2025-02-07 04:17:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:23 UTC)" (scheduled at 2025-02-07 04:17:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:17:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:17:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:17:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:17:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:40.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 04:17:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:54 UTC)" (scheduled at 2025-02-07 04:17:24.423055+00:00)
gcworker stdout | 2025-02-07 04:17:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:17:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:17:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:17:27,826 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:17:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:17:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:33.814166+00:00 (in 5.081555 seconds)
notificationworker stdout | 2025-02-07 04:17:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:38 UTC)" (scheduled at 2025-02-07 04:17:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:17:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:17:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 28, 732944), True, datetime.datetime(2025, 2, 7, 4, 17, 28, 732944), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:17:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:17:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:17:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:17:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:17:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:44.225378+00:00 (in 14.999532 seconds)
quotatotalworker stdout | 2025-02-07 04:17:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:44 UTC)" (scheduled at 2025-02-07 04:17:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:17:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:17:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:17:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:17:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:17:29,235 [235] [DEBUG] [app] Starting request: urn:request:821437fa-0049-4c9c-85d4-d37033776960 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:29,235 [238] [DEBUG] [app] Starting request: urn:request:ddc25166-3a63-444b-9304-ec3b30dbac4d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:17:29,239 [239] [DEBUG] [app] Starting request: urn:request:28a76914-4c68-4145-8716-6f72bf095db6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:29,239 [245] [DEBUG] [app] Starting request: urn:request:68d387a5-e36b-4d15-94a1-2a2dee6a0d27 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:29,240 [239] [DEBUG] [app] Ending request: urn:request:28a76914-4c68-4145-8716-6f72bf095db6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28a76914-4c68-4145-8716-6f72bf095db6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:17:29,240 [245] [DEBUG] [app] Ending request: urn:request:68d387a5-e36b-4d15-94a1-2a2dee6a0d27 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68d387a5-e36b-4d15-94a1-2a2dee6a0d27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:17:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:17:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:17:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:29,243 [238] [DEBUG] [app] Starting request: urn:request:0e640fc1-2cf2-4105-85ff-35d9dd1249b2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:29,243 [238] [DEBUG] [app] Ending request: urn:request:0e640fc1-2cf2-4105-85ff-35d9dd1249b2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0e640fc1-2cf2-4105-85ff-35d9dd1249b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:17:29,243 [237] [DEBUG] [app] Starting request: urn:request:03361ddd-0141-4c4f-a6df-6e37244fecf4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:17:29,243 [237] [DEBUG] [app] Ending request: urn:request:03361ddd-0141-4c4f-a6df-6e37244fecf4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03361ddd-0141-4c4f-a6df-6e37244fecf4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:17:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:17:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:17:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:17:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:29,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:29,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:29,256 [235] [DEBUG] [app] Ending request: urn:request:821437fa-0049-4c9c-85d4-d37033776960 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:821437fa-0049-4c9c-85d4-d37033776960', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:29,256 [238] [DEBUG] [app] Ending request: urn:request:ddc25166-3a63-444b-9304-ec3b30dbac4d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ddc25166-3a63-444b-9304-ec3b30dbac4d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:17:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
repositoryactioncounter stdout | 2025-02-07 04:17:29,695 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:17:30,782 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:17:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:17:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:32.625945+00:00 (in 1.004289 seconds)
namespacegcworker stdout | 2025-02-07 04:17:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:31 UTC)" (scheduled at 2025-02-07 04:17:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:17:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:17:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 31, 621940), True, datetime.datetime(2025, 2, 7, 4, 17, 31, 621940), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:17:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:17:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:17:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:17:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:17:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:31.621131+00:00 (in 58.994718 seconds)
namespacegcworker stdout | 2025-02-07 04:17:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:32 UTC)" (scheduled at 2025-02-07 04:17:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:17:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:17:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:17:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:17:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:38.732147+00:00 (in 4.917547 seconds)
notificationworker stdout | 2025-02-07 04:17:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:22:33 UTC)" (scheduled at 2025-02-07 04:17:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:17:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 17, 33, 814866), 'notification/%'])
notificationworker stdout | 2025-02-07 04:17:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 17, 33, 814866), True, datetime.datetime(2025, 2, 7, 4, 17, 33, 814866), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:17:33,824 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 17, 33, 814866), True, datetime.datetime(2025, 2, 7, 4, 17, 33, 814866), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 17, 33, 814866), 'notification/%'])
notificationworker stdout | 2025-02-07 04:17:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:17:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:22:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:17:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:17:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:03.919286+00:00 (in 29.999583 seconds)
buildlogsarchiver stdout | 2025-02-07 04:17:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:03 UTC)" (scheduled at 2025-02-07 04:17:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:17:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 17, 33, 919945), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:17:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:17:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:17:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:17:35,150 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:17:36,202 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:56.713547+00:00 (in 18.997200 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:37 UTC)" (scheduled at 2025-02-07 04:17:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:17:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:17:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:48.732147+00:00 (in 9.999449 seconds)
notificationworker stdout | 2025-02-07 04:17:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:48 UTC)" (scheduled at 2025-02-07 04:17:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:17:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:17:38,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 38, 733129), True, datetime.datetime(2025, 2, 7, 4, 17, 38, 733129), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:17:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:17:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:17:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:17:40,062 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:17:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:17:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:54.423055+00:00 (in 13.998388 seconds)
gcworker stdout | 2025-02-07 04:17:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:10 UTC)" (scheduled at 2025-02-07 04:17:40.424250+00:00)
gcworker stdout | 2025-02-07 04:17:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:17:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901560430, None, 1, 0])
gcworker stdout | 2025-02-07 04:17:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:17:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:17:40,767 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:17:41,259 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:17:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:17:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:11.831521+00:00 (in 29.999515 seconds)
autopruneworker stdout | 2025-02-07 04:17:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:11 UTC)" (scheduled at 2025-02-07 04:17:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:17:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898261836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:17:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:17:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:17:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:17:42,528 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:17:43,437 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:17:43,492 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:17:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:17:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:59.225378+00:00 (in 14.999529 seconds)
quotatotalworker stdout | 2025-02-07 04:17:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:59 UTC)" (scheduled at 2025-02-07 04:17:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:17:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:17:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:17:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:17:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:17:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:17:44,236 [235] [DEBUG] [app] Starting request: urn:request:44e77c66-7c50-473d-9b5a-f3a574fe2b8c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:44,236 [238] [DEBUG] [app] Starting request: urn:request:8abe20d4-3dd2-4c98-b399-afc8c48a0198 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:17:44,239 [239] [DEBUG] [app] Starting request: urn:request:2857f324-3437-4774-9241-24bd9612f658 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:44,239 [239] [DEBUG] [app] Ending request: urn:request:2857f324-3437-4774-9241-24bd9612f658 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2857f324-3437-4774-9241-24bd9612f658', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:17:44,239 [245] [DEBUG] [app] Starting request: urn:request:c2babe52-6d5d-45e9-8b49-71203cfe95c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:17:44,240 [245] [DEBUG] [app] Ending request: urn:request:c2babe52-6d5d-45e9-8b49-71203cfe95c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2babe52-6d5d-45e9-8b49-71203cfe95c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:17:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:17:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:17:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:44,242 [235] [DEBUG] [app] Starting request: urn:request:17c30d96-9802-461e-90a2-e9c60a488362 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:44,243 [235] [DEBUG] [app] Ending request: urn:request:17c30d96-9802-461e-90a2-e9c60a488362 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17c30d96-9802-461e-90a2-e9c60a488362', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:17:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:17:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:44,244 [237] [DEBUG] [app] Starting request: urn:request:528bddc3-3508-4d57-ae83-1917e2eee215 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:17:44,244 [237] [DEBUG] [app] Ending request: urn:request:528bddc3-3508-4d57-ae83-1917e2eee215 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:528bddc3-3508-4d57-ae83-1917e2eee215', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:17:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:17:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:17:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:44,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:44,255 [235] [DEBUG] [app] Ending request: urn:request:44e77c66-7c50-473d-9b5a-f3a574fe2b8c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44e77c66-7c50-473d-9b5a-f3a574fe2b8c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:17:44,256 [238] [DEBUG] [app] Ending request: urn:request:8abe20d4-3dd2-4c98-b399-afc8c48a0198 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8abe20d4-3dd2-4c98-b399-afc8c48a0198', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:44,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
autopruneworker stdout | 2025-02-07 04:17:44,578 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:44,965 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:17:45,565 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:17:45,606 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:17:45,746 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:17:46,079 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:17:46,568 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:17:46,571 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:17:46,574 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:17:46,578 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:17:46,581 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:17:46,591 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:17:46,658 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:17:46,678 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:17:47,594 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:17:47,597 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:17:47,600 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:17:48,343 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:17:48,490 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:17:48,723 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:17:48,729 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
notificationworker stdout | 2025-02-07 04:17:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gunicorn-registry stdout | 2025-02-07 04:17:48,732 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 04:17:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:50.814677+00:00 (in 2.082044 seconds)
notificationworker stdout | 2025-02-07 04:17:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:58 UTC)" (scheduled at 2025-02-07 04:17:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:17:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:17:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 48, 732926), True, datetime.datetime(2025, 2, 7, 4, 17, 48, 732926), 0, 'notification/%', 50, 1, 0])
gunicorn-registry stdout | 2025-02-07 04:17:48,735 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
notificationworker stdout | 2025-02-07 04:17:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:17:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:17:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:17:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:17:48,742 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:17:48,745 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:17:48,748 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:17:48,752 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:17:48,755 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
blobuploadcleanupworker stdout | 2025-02-07 04:17:48,810 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:17:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:17:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:17:58.732147+00:00 (in 7.916981 seconds)
notificationworker stdout | 2025-02-07 04:17:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:50 UTC)" (scheduled at 2025-02-07 04:17:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:17:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:17:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:17:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:17:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:10.424250+00:00 (in 16.000754 seconds)
gcworker stdout | 2025-02-07 04:17:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:24 UTC)" (scheduled at 2025-02-07 04:17:54.423055+00:00)
gcworker stdout | 2025-02-07 04:17:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:17:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:17:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:17:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:56.118855+00:00 (in 59.999476 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:17:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:56 UTC)" (scheduled at 2025-02-07 04:17:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:17:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:17:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:17:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:37.715878+00:00 (in 41.001881 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:56 UTC)" (scheduled at 2025-02-07 04:17:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:17:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:17:57,862 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:17:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:17:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:08.732147+00:00 (in 9.999445 seconds)
notificationworker stdout | 2025-02-07 04:17:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:08 UTC)" (scheduled at 2025-02-07 04:17:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:17:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:17:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 17, 58, 733017), True, datetime.datetime(2025, 2, 7, 4, 17, 58, 733017), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:17:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:17:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:17:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:17:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:17:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:14.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 04:17:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:14 UTC)" (scheduled at 2025-02-07 04:17:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:17:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:17:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:17:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:17:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:17:59,235 [238] [DEBUG] [app] Starting request: urn:request:86283a29-a1eb-4358-be11-3496b06c8bc1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:59,235 [237] [DEBUG] [app] Starting request: urn:request:43339b5d-2c6b-4ffb-9823-d11a909d2b77 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:17:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:17:59,238 [246] [DEBUG] [app] Starting request: urn:request:2ca9ba08-e0d2-4a7d-895c-faba1e30ddd6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:59,238 [244] [DEBUG] [app] Starting request: urn:request:3062ac0c-cbf6-4ff0-9225-93c4507b9626 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:17:59,239 [246] [DEBUG] [app] Ending request: urn:request:2ca9ba08-e0d2-4a7d-895c-faba1e30ddd6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ca9ba08-e0d2-4a7d-895c-faba1e30ddd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:17:59,239 [244] [DEBUG] [app] Ending request: urn:request:3062ac0c-cbf6-4ff0-9225-93c4507b9626 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3062ac0c-cbf6-4ff0-9225-93c4507b9626', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:17:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:17:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:17:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:17:59,242 [238] [DEBUG] [app] Starting request: urn:request:ad36235c-60f5-4c7f-94e4-01bd220aad9d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:59,242 [238] [DEBUG] [app] Ending request: urn:request:ad36235c-60f5-4c7f-94e4-01bd220aad9d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad36235c-60f5-4c7f-94e4-01bd220aad9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:17:59,242 [237] [DEBUG] [app] Starting request: urn:request:943ca46a-f5fc-4bc8-9ac3-110a71ebed0c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:17:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:17:59,243 [237] [DEBUG] [app] Ending request: urn:request:943ca46a-f5fc-4bc8-9ac3-110a71ebed0c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:943ca46a-f5fc-4bc8-9ac3-110a71ebed0c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:17:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:17:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:17:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:17:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:17:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:17:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:17:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:17:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:17:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:17:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:17:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:17:59,254 [237] [DEBUG] [app] Ending request: urn:request:43339b5d-2c6b-4ffb-9823-d11a909d2b77 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43339b5d-2c6b-4ffb-9823-d11a909d2b77', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:59,254 [238] [DEBUG] [app] Ending request: urn:request:86283a29-a1eb-4358-be11-3496b06c8bc1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:86283a29-a1eb-4358-be11-3496b06c8bc1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:17:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:17:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:17:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:17:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:17:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
repositoryactioncounter stdout | 2025-02-07 04:17:59,730 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:17:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:17:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:04.936688+00:00 (in 4.998315 seconds)
exportactionlogsworker stdout | 2025-02-07 04:17:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:59 UTC)" (scheduled at 2025-02-07 04:17:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:17:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:17:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:18:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:18:00,818 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:18:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:18:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:23.420886+00:00 (in 20.001309 seconds)
repositorygcworker stdout | 2025-02-07 04:18:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:03 UTC)" (scheduled at 2025-02-07 04:18:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:18:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:18:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 3, 419891), True, datetime.datetime(2025, 2, 7, 4, 18, 3, 419891), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:18:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:18:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:18:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:18:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:18:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:33.919286+00:00 (in 29.999561 seconds)
buildlogsarchiver stdout | 2025-02-07 04:18:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:33 UTC)" (scheduled at 2025-02-07 04:18:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:18:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 18, 3, 919977), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:18:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:18:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:18:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:18:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:18:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:59.937899+00:00 (in 55.000762 seconds)
exportactionlogsworker stdout | 2025-02-07 04:18:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:04 UTC)" (scheduled at 2025-02-07 04:18:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:18:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:18:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 4, 937432), True, datetime.datetime(2025, 2, 7, 4, 18, 4, 937432), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:18:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:18:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:18:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:18:05,186 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:18:06,234 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:18:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:18:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:18.732147+00:00 (in 9.999524 seconds)
notificationworker stdout | 2025-02-07 04:18:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:18 UTC)" (scheduled at 2025-02-07 04:18:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:18:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:18:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 8, 732892), True, datetime.datetime(2025, 2, 7, 4, 18, 8, 732892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:18:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:18:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:18:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:18:10,078 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:18:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:18:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:24.423055+00:00 (in 13.998342 seconds)
gcworker stdout | 2025-02-07 04:18:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:40 UTC)" (scheduled at 2025-02-07 04:18:10.424250+00:00)
gcworker stdout | 2025-02-07 04:18:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:18:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901590431, None, 1, 0])
gcworker stdout | 2025-02-07 04:18:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:18:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:18:10,797 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:18:11,291 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:18:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:18:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:41.831521+00:00 (in 29.999500 seconds)
autopruneworker stdout | 2025-02-07 04:18:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:41 UTC)" (scheduled at 2025-02-07 04:18:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:18:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898291836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:18:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:18:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:18:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:18:12,565 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:18:13,469 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:18:13,528 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:18:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:18:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:29.225378+00:00 (in 14.999555 seconds)
quotatotalworker stdout | 2025-02-07 04:18:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:29 UTC)" (scheduled at 2025-02-07 04:18:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:18:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:18:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:18:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:18:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:18:14,235 [235] [DEBUG] [app] Starting request: urn:request:9efd77c6-f0c9-4750-a6d0-2ff991ff24a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:14,235 [237] [DEBUG] [app] Starting request: urn:request:084dd4da-f434-4e2d-847c-434d6cba371e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:18:14,238 [239] [DEBUG] [app] Starting request: urn:request:ad9e8419-625e-445e-9f97-b33ca5e0ec7e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:14,238 [239] [DEBUG] [app] Ending request: urn:request:ad9e8419-625e-445e-9f97-b33ca5e0ec7e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ad9e8419-625e-445e-9f97-b33ca5e0ec7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:18:14,238 [244] [DEBUG] [app] Starting request: urn:request:2561f511-d26d-467e-b678-c915a2db5083 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:14,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:18:14,239 [244] [DEBUG] [app] Ending request: urn:request:2561f511-d26d-467e-b678-c915a2db5083 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2561f511-d26d-467e-b678-c915a2db5083', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:18:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:18:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:18:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:14,241 [238] [DEBUG] [app] Starting request: urn:request:75e2bf4d-32f0-472a-bee9-24ca7ce3fc3a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:14,241 [238] [DEBUG] [app] Ending request: urn:request:75e2bf4d-32f0-472a-bee9-24ca7ce3fc3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:75e2bf4d-32f0-472a-bee9-24ca7ce3fc3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:18:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:18:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:14,242 [236] [DEBUG] [app] Starting request: urn:request:e17ebf1f-7a52-4431-bca3-fe68b086c42b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:14,242 [236] [DEBUG] [app] Ending request: urn:request:e17ebf1f-7a52-4431-bca3-fe68b086c42b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e17ebf1f-7a52-4431-bca3-fe68b086c42b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:18:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:18:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:14,253 [237] [DEBUG] [app] Ending request: urn:request:084dd4da-f434-4e2d-847c-434d6cba371e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:084dd4da-f434-4e2d-847c-434d6cba371e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:18:14,254 [235] [DEBUG] [app] Ending request: urn:request:9efd77c6-f0c9-4750-a6d0-2ff991ff24a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9efd77c6-f0c9-4750-a6d0-2ff991ff24a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
autopruneworker stdout | 2025-02-07 04:18:14,595 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:15,002 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:18:15,579 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:18:15,640 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:18:15,774 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:18:16,089 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:18:16,575 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:18:16,579 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:18:16,582 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:18:16,585 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:18:16,588 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:18:16,627 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:18:16,685 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:18:16,714 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:18:17,602 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:18:17,605 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:18:17,607 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:18:18,380 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:18:18,522 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:18:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:18:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:28.732147+00:00 (in 9.999541 seconds)
notificationworker stdout | 2025-02-07 04:18:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:28 UTC)" (scheduled at 2025-02-07 04:18:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:18:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:18:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 18, 732917), True, datetime.datetime(2025, 2, 7, 4, 18, 18, 732917), 0, 'notification/%', 50, 1, 0])
gunicorn-registry stdout | 2025-02-07 04:18:18,738 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
notificationworker stdout | 2025-02-07 04:18:18,741 [73] [DEBUG] [workers.queueworker] No more work.
gunicorn-registry stdout | 2025-02-07 04:18:18,741 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
notificationworker stdout | 2025-02-07 04:18:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:18:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:18:18,745 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:18:18,748 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:18:18,752 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:18:18,755 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:18:18,758 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:18:18,762 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:18:18,765 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
blobuploadcleanupworker stdout | 2025-02-07 04:18:18,847 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:18:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:18:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:32.420707+00:00 (in 8.999320 seconds)
repositorygcworker stdout | 2025-02-07 04:18:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:23 UTC)" (scheduled at 2025-02-07 04:18:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:18:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:18:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:18:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:18:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:40.424250+00:00 (in 16.000715 seconds)
gcworker stdout | 2025-02-07 04:18:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:54 UTC)" (scheduled at 2025-02-07 04:18:24.423055+00:00)
gcworker stdout | 2025-02-07 04:18:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:18:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:18:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:18:27,898 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:18:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:18:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:38.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 04:18:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:38 UTC)" (scheduled at 2025-02-07 04:18:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:18:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:18:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 28, 732862), True, datetime.datetime(2025, 2, 7, 4, 18, 28, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:18:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:18:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:18:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:18:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:18:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:44.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 04:18:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:44 UTC)" (scheduled at 2025-02-07 04:18:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:18:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:18:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:18:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:18:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:18:29,235 [237] [DEBUG] [app] Starting request: urn:request:11764898-6ec6-4308-a0ad-506d6fd9819a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:29,235 [238] [DEBUG] [app] Starting request: urn:request:d9fbba66-3e48-43e3-9c23-d1ca2c428792 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:18:29,239 [239] [DEBUG] [app] Starting request: urn:request:27710faf-5c5f-49b4-a7c0-7aa28aaac87b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:29,239 [245] [DEBUG] [app] Starting request: urn:request:626f11fb-91b4-475d-ab7b-66d9fd36f65c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:29,239 [245] [DEBUG] [app] Ending request: urn:request:626f11fb-91b4-475d-ab7b-66d9fd36f65c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:626f11fb-91b4-475d-ab7b-66d9fd36f65c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:18:29,239 [239] [DEBUG] [app] Ending request: urn:request:27710faf-5c5f-49b4-a7c0-7aa28aaac87b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27710faf-5c5f-49b4-a7c0-7aa28aaac87b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:18:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:18:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:18:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:29,243 [236] [DEBUG] [app] Starting request: urn:request:058f9170-dd6f-4f18-b4d9-0feee482ee42 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:29,243 [235] [DEBUG] [app] Starting request: urn:request:c5099b97-ac70-48c9-b69c-662ecc6feddc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:29,243 [236] [DEBUG] [app] Ending request: urn:request:058f9170-dd6f-4f18-b4d9-0feee482ee42 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:058f9170-dd6f-4f18-b4d9-0feee482ee42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:18:29,243 [235] [DEBUG] [app] Ending request: urn:request:c5099b97-ac70-48c9-b69c-662ecc6feddc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5099b97-ac70-48c9-b69c-662ecc6feddc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:18:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:18:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:18:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:29,255 [238] [DEBUG] [app] Ending request: urn:request:d9fbba66-3e48-43e3-9c23-d1ca2c428792 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9fbba66-3e48-43e3-9c23-d1ca2c428792', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:29,255 [237] [DEBUG] [app] Ending request: urn:request:11764898-6ec6-4308-a0ad-506d6fd9819a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11764898-6ec6-4308-a0ad-506d6fd9819a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:18:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:18:29,767 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:18:30,836 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:18:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:18:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:32.625945+00:00 (in 1.004312 seconds)
namespacegcworker stdout | 2025-02-07 04:18:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:31 UTC)" (scheduled at 2025-02-07 04:18:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:18:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:18:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 31, 621946), True, datetime.datetime(2025, 2, 7, 4, 18, 31, 621946), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:18:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:18:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:18:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:18:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:18:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:03.419110+00:00 (in 30.997961 seconds)
repositorygcworker stdout | 2025-02-07 04:18:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:23:32 UTC)" (scheduled at 2025-02-07 04:18:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:18:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 18, 32, 421433), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:18:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 18, 32, 421433), True, datetime.datetime(2025, 2, 7, 4, 18, 32, 421433), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:18:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 18, 32, 421433), True, datetime.datetime(2025, 2, 7, 4, 18, 32, 421433), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 18, 32, 421433), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:18:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:18:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:23:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:18:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:18:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:31.621131+00:00 (in 58.994654 seconds)
namespacegcworker stdout | 2025-02-07 04:18:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:32 UTC)" (scheduled at 2025-02-07 04:18:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:18:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:18:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:18:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:18:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:03.919286+00:00 (in 29.999535 seconds)
buildlogsarchiver stdout | 2025-02-07 04:18:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:03 UTC)" (scheduled at 2025-02-07 04:18:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:18:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 18, 33, 920007), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:18:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:18:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:18:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:18:35,206 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:18:36,271 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:56.713547+00:00 (in 18.997196 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:37 UTC)" (scheduled at 2025-02-07 04:18:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:18:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:18:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:48.732147+00:00 (in 9.999454 seconds)
notificationworker stdout | 2025-02-07 04:18:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:48 UTC)" (scheduled at 2025-02-07 04:18:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:18:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:18:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 38, 733085), True, datetime.datetime(2025, 2, 7, 4, 18, 38, 733085), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:18:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:18:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:18:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:18:40,089 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:18:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:18:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:54.423055+00:00 (in 13.998372 seconds)
gcworker stdout | 2025-02-07 04:18:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:10 UTC)" (scheduled at 2025-02-07 04:18:40.424250+00:00)
gcworker stdout | 2025-02-07 04:18:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:18:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901620431, None, 1, 0])
gcworker stdout | 2025-02-07 04:18:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:18:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:18:40,832 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:18:41,327 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:18:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:18:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:11.831521+00:00 (in 29.999558 seconds)
autopruneworker stdout | 2025-02-07 04:18:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:11 UTC)" (scheduled at 2025-02-07 04:18:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:18:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898321836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:18:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:18:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:18:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:18:42,572 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:18:43,506 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:18:43,565 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:18:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:18:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:59.225378+00:00 (in 14.999520 seconds)
quotatotalworker stdout | 2025-02-07 04:18:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:59 UTC)" (scheduled at 2025-02-07 04:18:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:18:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:18:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:18:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:18:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:18:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:18:44,236 [235] [DEBUG] [app] Starting request: urn:request:d0548a81-079d-4488-b74b-63517019110f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:44,236 [238] [DEBUG] [app] Starting request: urn:request:8a425656-dd2d-4228-a94b-f09beee604b3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:18:44,239 [239] [DEBUG] [app] Starting request: urn:request:af97530b-2b30-4ed8-be28-83bbfe076f44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:44,239 [244] [DEBUG] [app] Starting request: urn:request:bd23564d-0a9a-4df7-ad9e-12764babac0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:44,239 [239] [DEBUG] [app] Ending request: urn:request:af97530b-2b30-4ed8-be28-83bbfe076f44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af97530b-2b30-4ed8-be28-83bbfe076f44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:18:44,239 [244] [DEBUG] [app] Ending request: urn:request:bd23564d-0a9a-4df7-ad9e-12764babac0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd23564d-0a9a-4df7-ad9e-12764babac0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:18:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:18:44,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:18:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:44,242 [235] [DEBUG] [app] Starting request: urn:request:76f5275c-3487-43a4-bfea-ef4f178f6a62 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:44,242 [235] [DEBUG] [app] Ending request: urn:request:76f5275c-3487-43a4-bfea-ef4f178f6a62 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76f5275c-3487-43a4-bfea-ef4f178f6a62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:18:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:18:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:44,243 [237] [DEBUG] [app] Starting request: urn:request:a98a738e-1daf-4ba6-9f36-e4b5d5296fc5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:44,244 [237] [DEBUG] [app] Ending request: urn:request:a98a738e-1daf-4ba6-9f36-e4b5d5296fc5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a98a738e-1daf-4ba6-9f36-e4b5d5296fc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:18:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:18:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:44,255 [238] [DEBUG] [app] Ending request: urn:request:8a425656-dd2d-4228-a94b-f09beee604b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a425656-dd2d-4228-a94b-f09beee604b3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:44,255 [235] [DEBUG] [app] Ending request: urn:request:d0548a81-079d-4488-b74b-63517019110f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d0548a81-079d-4488-b74b-63517019110f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:18:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
autopruneworker stdout | 2025-02-07 04:18:44,631 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:45,021 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:18:45,616 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:18:45,677 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:18:45,806 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:18:46,126 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:18:46,583 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:18:46,586 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:18:46,590 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:18:46,593 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:18:46,595 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:18:46,646 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:18:46,718 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:18:46,730 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:18:47,609 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:18:47,612 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:18:47,615 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:18:48,406 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:18:48,544 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:18:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:18:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:50.814677+00:00 (in 2.082071 seconds)
notificationworker stdout | 2025-02-07 04:18:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:58 UTC)" (scheduled at 2025-02-07 04:18:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:18:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:18:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 48, 732878), True, datetime.datetime(2025, 2, 7, 4, 18, 48, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:18:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:18:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:18:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:18:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:18:48,750 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:18:48,752 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:18:48,755 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:18:48,759 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:18:48,762 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:18:48,764 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:18:48,767 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:18:48,770 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:18:48,774 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
blobuploadcleanupworker stdout | 2025-02-07 04:18:48,883 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:18:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:18:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:18:58.732147+00:00 (in 7.916988 seconds)
notificationworker stdout | 2025-02-07 04:18:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:50 UTC)" (scheduled at 2025-02-07 04:18:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:18:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:18:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:18:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:18:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:10.424250+00:00 (in 16.000717 seconds)
gcworker stdout | 2025-02-07 04:18:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:24 UTC)" (scheduled at 2025-02-07 04:18:54.423055+00:00)
gcworker stdout | 2025-02-07 04:18:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:18:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:18:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:18:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:56.118855+00:00 (in 59.999519 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:18:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:56 UTC)" (scheduled at 2025-02-07 04:18:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:18:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:18:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:18:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:37.715878+00:00 (in 41.001874 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:56 UTC)" (scheduled at 2025-02-07 04:18:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:18:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:18:57,931 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:18:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:18:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:08.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 04:18:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:08 UTC)" (scheduled at 2025-02-07 04:18:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:18:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:18:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 18, 58, 732885), True, datetime.datetime(2025, 2, 7, 4, 18, 58, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:18:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:18:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:18:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:18:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:18:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:14.225378+00:00 (in 14.999507 seconds)
quotatotalworker stdout | 2025-02-07 04:18:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:14 UTC)" (scheduled at 2025-02-07 04:18:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:18:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:18:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:18:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:18:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:18:59,235 [235] [DEBUG] [app] Starting request: urn:request:4f536c22-ced9-45ec-95b0-43bc8e805130 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:59,235 [236] [DEBUG] [app] Starting request: urn:request:7c3cf4b4-3976-48dd-8b30-0313b10a992e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:18:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:18:59,238 [246] [DEBUG] [app] Starting request: urn:request:65184c83-e90c-4106-9393-936030e76625 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:59,238 [239] [DEBUG] [app] Starting request: urn:request:915f3ad3-b61a-4423-bab2-814136de2b68 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:18:59,238 [239] [DEBUG] [app] Ending request: urn:request:915f3ad3-b61a-4423-bab2-814136de2b68 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:915f3ad3-b61a-4423-bab2-814136de2b68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:18:59,238 [246] [DEBUG] [app] Ending request: urn:request:65184c83-e90c-4106-9393-936030e76625 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:65184c83-e90c-4106-9393-936030e76625', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:18:59,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:18:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:18:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:18:59,241 [235] [DEBUG] [app] Starting request: urn:request:5259c134-cfb8-4352-9f0e-2a473bccebcb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:59,241 [238] [DEBUG] [app] Starting request: urn:request:f530eed5-908f-44e5-8106-5b116f2a0eff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:18:59,241 [235] [DEBUG] [app] Ending request: urn:request:5259c134-cfb8-4352-9f0e-2a473bccebcb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5259c134-cfb8-4352-9f0e-2a473bccebcb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:18:59,241 [238] [DEBUG] [app] Ending request: urn:request:f530eed5-908f-44e5-8106-5b116f2a0eff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f530eed5-908f-44e5-8106-5b116f2a0eff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:18:59,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:18:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:18:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:18:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:18:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:18:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:59,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:18:59,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:18:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:18:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:18:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:59,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:18:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:18:59,252 [235] [DEBUG] [app] Ending request: urn:request:4f536c22-ced9-45ec-95b0-43bc8e805130 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f536c22-ced9-45ec-95b0-43bc8e805130', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:59,253 [236] [DEBUG] [app] Ending request: urn:request:7c3cf4b4-3976-48dd-8b30-0313b10a992e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c3cf4b4-3976-48dd-8b30-0313b10a992e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:18:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:18:59,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:18:59,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:18:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:18:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:18:59,774 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:18:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:18:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:04.936688+00:00 (in 4.998361 seconds)
exportactionlogsworker stdout | 2025-02-07 04:18:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:59 UTC)" (scheduled at 2025-02-07 04:18:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:18:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:18:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:19:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:19:00,853 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:19:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:19:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:23.420886+00:00 (in 20.001319 seconds)
repositorygcworker stdout | 2025-02-07 04:19:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:03 UTC)" (scheduled at 2025-02-07 04:19:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:19:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:19:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 3, 419841), True, datetime.datetime(2025, 2, 7, 4, 19, 3, 419841), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:19:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:19:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:19:03,426 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:19:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:19:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:33.919286+00:00 (in 29.999490 seconds)
buildlogsarchiver stdout | 2025-02-07 04:19:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:33 UTC)" (scheduled at 2025-02-07 04:19:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:19:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 19, 3, 920079), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:19:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:19:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:19:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:19:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:19:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:52.937648+00:00 (in 48.000534 seconds)
exportactionlogsworker stdout | 2025-02-07 04:19:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:04 UTC)" (scheduled at 2025-02-07 04:19:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:19:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:19:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 4, 937419), True, datetime.datetime(2025, 2, 7, 4, 19, 4, 937419), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:19:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:19:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:19:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:19:05,229 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:19:06,304 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:19:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:19:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:18.732147+00:00 (in 9.999430 seconds)
notificationworker stdout | 2025-02-07 04:19:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:18 UTC)" (scheduled at 2025-02-07 04:19:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:19:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:19:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 8, 733099), True, datetime.datetime(2025, 2, 7, 4, 19, 8, 733099), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:19:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:19:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:19:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:19:10,126 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:19:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:19:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:24.423055+00:00 (in 13.998379 seconds)
gcworker stdout | 2025-02-07 04:19:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:40 UTC)" (scheduled at 2025-02-07 04:19:10.424250+00:00)
gcworker stdout | 2025-02-07 04:19:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:19:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901650430, None, 1, 0])
gcworker stdout | 2025-02-07 04:19:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:19:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:19:10,839 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:19:11,345 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:19:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:19:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:41.831521+00:00 (in 29.999540 seconds)
autopruneworker stdout | 2025-02-07 04:19:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:41 UTC)" (scheduled at 2025-02-07 04:19:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:19:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898351836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:19:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:19:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:19:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:19:12,608 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:19:13,536 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:19:13,601 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:19:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:19:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:29.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:19:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:29 UTC)" (scheduled at 2025-02-07 04:19:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:19:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:19:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:19:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:19:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:19:14,235 [237] [DEBUG] [app] Starting request: urn:request:b543ff44-471d-4001-ae55-143a68e73d0d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:14,235 [235] [DEBUG] [app] Starting request: urn:request:54e26568-7c5f-401f-90ec-c62ff8221b51 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:19:14,239 [239] [DEBUG] [app] Starting request: urn:request:0f2fd49a-e666-4b04-8fcd-5d84148ee62c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:14,239 [245] [DEBUG] [app] Starting request: urn:request:ccc914ac-cd53-47e0-a95f-c1b2c907a2f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:14,239 [245] [DEBUG] [app] Ending request: urn:request:ccc914ac-cd53-47e0-a95f-c1b2c907a2f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ccc914ac-cd53-47e0-a95f-c1b2c907a2f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:19:14,239 [239] [DEBUG] [app] Ending request: urn:request:0f2fd49a-e666-4b04-8fcd-5d84148ee62c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f2fd49a-e666-4b04-8fcd-5d84148ee62c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:19:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:19:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:19:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:14,242 [235] [DEBUG] [app] Starting request: urn:request:572edab1-a3fb-4820-ab25-71dc2ba7d199 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:19:14,242 [235] [DEBUG] [app] Ending request: urn:request:572edab1-a3fb-4820-ab25-71dc2ba7d199 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:572edab1-a3fb-4820-ab25-71dc2ba7d199', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:19:14,242 [238] [DEBUG] [app] Starting request: urn:request:7017f5d1-e0c1-4b86-adee-b7766a28d6f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:19:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:14,242 [238] [DEBUG] [app] Ending request: urn:request:7017f5d1-e0c1-4b86-adee-b7766a28d6f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7017f5d1-e0c1-4b86-adee-b7766a28d6f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:19:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:19:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:14,254 [235] [DEBUG] [app] Ending request: urn:request:54e26568-7c5f-401f-90ec-c62ff8221b51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:54e26568-7c5f-401f-90ec-c62ff8221b51', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:14,254 [237] [DEBUG] [app] Ending request: urn:request:b543ff44-471d-4001-ae55-143a68e73d0d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b543ff44-471d-4001-ae55-143a68e73d0d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:19:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:19:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:19:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
autopruneworker stdout | 2025-02-07 04:19:14,667 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:15,045 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:19:15,650 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:19:15,712 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:19:15,831 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:19:16,154 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:19:16,591 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:19:16,594 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:19:16,597 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:19:16,600 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:19:16,603 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:19:16,682 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:19:16,730 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:19:16,758 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:19:17,617 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:19:17,620 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:19:17,623 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:19:18,437 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:19:18,571 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:19:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:19:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:28.732147+00:00 (in 9.999503 seconds)
notificationworker stdout | 2025-02-07 04:19:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:28 UTC)" (scheduled at 2025-02-07 04:19:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:19:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:19:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 18, 732934), True, datetime.datetime(2025, 2, 7, 4, 19, 18, 732934), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:19:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:19:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:19:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:19:18,761 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:19:18,764 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:19:18,769 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:19:18,773 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:19:18,776 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:19:18,779 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:19:18,782 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:19:18,784 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:19:18,787 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
blobuploadcleanupworker stdout | 2025-02-07 04:19:18,920 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:19:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:19:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:03.419110+00:00 (in 39.997732 seconds)
repositorygcworker stdout | 2025-02-07 04:19:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:23 UTC)" (scheduled at 2025-02-07 04:19:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:19:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:19:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:19:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:19:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:40.424250+00:00 (in 16.000692 seconds)
gcworker stdout | 2025-02-07 04:19:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:54 UTC)" (scheduled at 2025-02-07 04:19:24.423055+00:00)
gcworker stdout | 2025-02-07 04:19:24,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 04:19:24,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:19:24,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:19:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:19:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:19:27,962 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:19:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:19:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:38.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 04:19:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:38 UTC)" (scheduled at 2025-02-07 04:19:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:19:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:19:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 28, 732919), True, datetime.datetime(2025, 2, 7, 4, 19, 28, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:19:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:19:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:19:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:19:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:19:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:44.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 04:19:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:44 UTC)" (scheduled at 2025-02-07 04:19:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:19:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:19:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:19:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:19:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:19:29,235 [238] [DEBUG] [app] Starting request: urn:request:b2a2818e-98eb-40da-bad1-f7cdaeddf160 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:29,235 [236] [DEBUG] [app] Starting request: urn:request:e36326b2-0b7e-40ca-a6d2-567dbb96b6f1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:19:29,238 [239] [DEBUG] [app] Starting request: urn:request:1a1af74b-6169-4cb1-a749-54d01c8ca9d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:29,238 [246] [DEBUG] [app] Starting request: urn:request:8edbabe2-154e-42e3-9c6b-b9422cc0d2ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:29,239 [246] [DEBUG] [app] Ending request: urn:request:8edbabe2-154e-42e3-9c6b-b9422cc0d2ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8edbabe2-154e-42e3-9c6b-b9422cc0d2ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:19:29,239 [239] [DEBUG] [app] Ending request: urn:request:1a1af74b-6169-4cb1-a749-54d01c8ca9d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1a1af74b-6169-4cb1-a749-54d01c8ca9d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:19:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:19:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:19:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:29,242 [235] [DEBUG] [app] Starting request: urn:request:e6106673-09f1-4e78-9bb7-8d956977d88e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:19:29,242 [235] [DEBUG] [app] Ending request: urn:request:e6106673-09f1-4e78-9bb7-8d956977d88e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6106673-09f1-4e78-9bb7-8d956977d88e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:19:29,242 [238] [DEBUG] [app] Starting request: urn:request:aef37d3f-05b2-4a81-bb99-617efbde5930 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:19:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:29,242 [238] [DEBUG] [app] Ending request: urn:request:aef37d3f-05b2-4a81-bb99-617efbde5930 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aef37d3f-05b2-4a81-bb99-617efbde5930', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:19:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:29,254 [236] [DEBUG] [app] Ending request: urn:request:e36326b2-0b7e-40ca-a6d2-567dbb96b6f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e36326b2-0b7e-40ca-a6d2-567dbb96b6f1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:29,254 [238] [DEBUG] [app] Ending request: urn:request:b2a2818e-98eb-40da-bad1-f7cdaeddf160 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2a2818e-98eb-40da-bad1-f7cdaeddf160', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:19:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:19:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
gunicorn-web stdout | 2025-02-07 04:19:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:19:29,810 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:19:30,889 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:19:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:19:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:32.625945+00:00 (in 1.004307 seconds)
namespacegcworker stdout | 2025-02-07 04:19:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:31 UTC)" (scheduled at 2025-02-07 04:19:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:19:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:19:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 31, 621941), True, datetime.datetime(2025, 2, 7, 4, 19, 31, 621941), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:19:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:19:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:19:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:19:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:19:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:31.621131+00:00 (in 58.994729 seconds)
namespacegcworker stdout | 2025-02-07 04:19:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:32 UTC)" (scheduled at 2025-02-07 04:19:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:19:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:19:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:19:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:19:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:03.919286+00:00 (in 29.999533 seconds)
buildlogsarchiver stdout | 2025-02-07 04:19:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:03 UTC)" (scheduled at 2025-02-07 04:19:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:19:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 19, 33, 920008), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:19:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:19:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:19:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:19:35,265 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:19:36,341 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:56.713547+00:00 (in 18.997179 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:37 UTC)" (scheduled at 2025-02-07 04:19:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:19:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:19:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:48.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:19:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:48 UTC)" (scheduled at 2025-02-07 04:19:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:19:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:19:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 38, 732860), True, datetime.datetime(2025, 2, 7, 4, 19, 38, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:19:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:19:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:19:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:19:40,162 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:19:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:19:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:54.423055+00:00 (in 13.998254 seconds)
gcworker stdout | 2025-02-07 04:19:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:10 UTC)" (scheduled at 2025-02-07 04:19:40.424250+00:00)
gcworker stdout | 2025-02-07 04:19:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:19:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901680431, None, 1, 0])
gcworker stdout | 2025-02-07 04:19:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:19:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:19:40,854 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:19:41,353 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:19:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:19:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:11.831521+00:00 (in 29.999550 seconds)
autopruneworker stdout | 2025-02-07 04:19:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:11 UTC)" (scheduled at 2025-02-07 04:19:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:19:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898381836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:19:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:19:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:19:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:19:42,617 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:19:43,573 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:19:43,630 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:19:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:19:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:59.225378+00:00 (in 14.999614 seconds)
quotatotalworker stdout | 2025-02-07 04:19:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:59 UTC)" (scheduled at 2025-02-07 04:19:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:19:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:19:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:19:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:19:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:19:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:19:44,235 [237] [DEBUG] [app] Starting request: urn:request:fb302f31-221f-48aa-9d1f-aaa867056064 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:44,235 [235] [DEBUG] [app] Starting request: urn:request:e2eb722c-32b9-4254-8b0d-73707e44c580 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:19:44,238 [239] [DEBUG] [app] Starting request: urn:request:820cdb80-e796-4046-bc76-7bd2dc7accea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:44,238 [245] [DEBUG] [app] Starting request: urn:request:324141d7-f895-4695-a0ee-8054a90b2c55 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:44,238 [245] [DEBUG] [app] Ending request: urn:request:324141d7-f895-4695-a0ee-8054a90b2c55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:324141d7-f895-4695-a0ee-8054a90b2c55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:19:44,238 [239] [DEBUG] [app] Ending request: urn:request:820cdb80-e796-4046-bc76-7bd2dc7accea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:820cdb80-e796-4046-bc76-7bd2dc7accea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:19:44,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:19:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:19:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:44,241 [235] [DEBUG] [app] Starting request: urn:request:552927cf-aa4a-479d-b3f8-1dbdc2ff7522 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:19:44,241 [236] [DEBUG] [app] Starting request: urn:request:f448c969-50bd-4332-be0b-c7ea5a4feb96 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:19:44,241 [235] [DEBUG] [app] Ending request: urn:request:552927cf-aa4a-479d-b3f8-1dbdc2ff7522 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:552927cf-aa4a-479d-b3f8-1dbdc2ff7522', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:19:44,241 [236] [DEBUG] [app] Ending request: urn:request:f448c969-50bd-4332-be0b-c7ea5a4feb96 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f448c969-50bd-4332-be0b-c7ea5a4feb96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:19:44,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:44,241 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:19:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:44,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:44,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:44,252 [235] [DEBUG] [app] Ending request: urn:request:e2eb722c-32b9-4254-8b0d-73707e44c580 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e2eb722c-32b9-4254-8b0d-73707e44c580', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:44,252 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:19:44,252 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:19:44,253 [237] [DEBUG] [app] Ending request: urn:request:fb302f31-221f-48aa-9d1f-aaa867056064 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb302f31-221f-48aa-9d1f-aaa867056064', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:19:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:19:44,703 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:45,081 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:19:45,665 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:19:45,730 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:19:45,843 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:19:46,190 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:19:46,599 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:19:46,602 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:19:46,605 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:19:46,607 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:19:46,610 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:19:46,698 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:19:46,747 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:19:46,774 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:19:47,625 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:19:47,629 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:19:47,632 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:19:48,473 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:19:48,606 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:19:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:19:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:50.814677+00:00 (in 2.082095 seconds)
notificationworker stdout | 2025-02-07 04:19:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:58 UTC)" (scheduled at 2025-02-07 04:19:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:19:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:19:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 48, 732863), True, datetime.datetime(2025, 2, 7, 4, 19, 48, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:19:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:19:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:19:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:19:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:19:48,769 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:19:48,772 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:19:48,776 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:19:48,782 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:19:48,785 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:19:48,788 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:19:48,791 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:19:48,793 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:19:48,796 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
blobuploadcleanupworker stdout | 2025-02-07 04:19:48,949 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:19:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:19:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:58.732147+00:00 (in 7.917000 seconds)
notificationworker stdout | 2025-02-07 04:19:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:50 UTC)" (scheduled at 2025-02-07 04:19:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:19:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:19:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:19:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:19:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:19:59.937899+00:00 (in 6.999813 seconds)
exportactionlogsworker stdout | 2025-02-07 04:19:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:24:52 UTC)" (scheduled at 2025-02-07 04:19:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:19:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 19, 52, 938343), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:19:52,944 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 19, 52, 938343), True, datetime.datetime(2025, 2, 7, 4, 19, 52, 938343), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:19:52,946 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 19, 52, 938343), True, datetime.datetime(2025, 2, 7, 4, 19, 52, 938343), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 19, 52, 938343), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:19:52,947 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:19:52,947 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:24:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:19:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:19:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:10.424250+00:00 (in 16.000727 seconds)
gcworker stdout | 2025-02-07 04:19:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:24 UTC)" (scheduled at 2025-02-07 04:19:54.423055+00:00)
gcworker stdout | 2025-02-07 04:19:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:19:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:19:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:19:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:56.118855+00:00 (in 59.999485 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:19:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:56 UTC)" (scheduled at 2025-02-07 04:19:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:19:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:19:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:19:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:37.715878+00:00 (in 41.001865 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:56 UTC)" (scheduled at 2025-02-07 04:19:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:19:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:19:57,999 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:19:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:19:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:08.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 04:19:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:08 UTC)" (scheduled at 2025-02-07 04:19:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:19:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:19:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 19, 58, 732880), True, datetime.datetime(2025, 2, 7, 4, 19, 58, 732880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:19:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:19:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:19:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:19:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:19:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:14.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 04:19:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:14 UTC)" (scheduled at 2025-02-07 04:19:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:19:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:19:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:19:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:19:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:19:59,236 [238] [DEBUG] [app] Starting request: urn:request:65e9f003-6fc9-459f-a265-4314acfd8f95 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:59,236 [235] [DEBUG] [app] Starting request: urn:request:d722b5ed-97b7-4911-b3b9-b7f84ee0afec (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:19:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:19:59,238 [239] [DEBUG] [app] Starting request: urn:request:2a405076-f32d-41d4-82a6-e76932f87150 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:59,239 [245] [DEBUG] [app] Starting request: urn:request:6570aecc-ed57-4fd7-8083-e4398e27fc61 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:19:59,239 [239] [DEBUG] [app] Ending request: urn:request:2a405076-f32d-41d4-82a6-e76932f87150 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a405076-f32d-41d4-82a6-e76932f87150', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:19:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:19:59,239 [245] [DEBUG] [app] Ending request: urn:request:6570aecc-ed57-4fd7-8083-e4398e27fc61 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6570aecc-ed57-4fd7-8083-e4398e27fc61', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:19:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:19:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:19:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:19:59,242 [235] [DEBUG] [app] Starting request: urn:request:87d69d65-22f3-475b-ab7a-d962e48b30cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:19:59,242 [235] [DEBUG] [app] Ending request: urn:request:87d69d65-22f3-475b-ab7a-d962e48b30cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87d69d65-22f3-475b-ab7a-d962e48b30cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:19:59,242 [237] [DEBUG] [app] Starting request: urn:request:6287791e-b944-4f05-8b2a-c86c396fee86 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:19:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:19:59,242 [237] [DEBUG] [app] Ending request: urn:request:6287791e-b944-4f05-8b2a-c86c396fee86 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6287791e-b944-4f05-8b2a-c86c396fee86', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:19:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:19:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:19:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:19:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:19:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:19:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:19:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:19:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:19:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:19:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:19:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:19:59,253 [235] [DEBUG] [app] Ending request: urn:request:d722b5ed-97b7-4911-b3b9-b7f84ee0afec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d722b5ed-97b7-4911-b3b9-b7f84ee0afec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:19:59,254 [238] [DEBUG] [app] Ending request: urn:request:65e9f003-6fc9-459f-a265-4314acfd8f95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:65e9f003-6fc9-459f-a265-4314acfd8f95', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:19:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:19:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:19:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:19:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:19:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:19:59,847 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:19:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:19:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:04.936688+00:00 (in 4.998350 seconds)
exportactionlogsworker stdout | 2025-02-07 04:19:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:59 UTC)" (scheduled at 2025-02-07 04:19:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:19:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:19:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:20:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:20:00,925 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:20:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:20:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:23.420886+00:00 (in 20.001264 seconds)
repositorygcworker stdout | 2025-02-07 04:20:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:03 UTC)" (scheduled at 2025-02-07 04:20:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:20:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:20:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 3, 419935), True, datetime.datetime(2025, 2, 7, 4, 20, 3, 419935), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:20:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:20:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:20:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:20:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:20:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:33.919286+00:00 (in 29.999537 seconds)
buildlogsarchiver stdout | 2025-02-07 04:20:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:33 UTC)" (scheduled at 2025-02-07 04:20:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:20:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 20, 3, 920001), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:20:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:20:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:20:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:20:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:20:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:59.937899+00:00 (in 55.000802 seconds)
exportactionlogsworker stdout | 2025-02-07 04:20:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:04 UTC)" (scheduled at 2025-02-07 04:20:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:20:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:20:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 4, 937383), True, datetime.datetime(2025, 2, 7, 4, 20, 4, 937383), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:20:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:20:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:20:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:20:05,284 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:20:06,354 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:20:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:20:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:18.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 04:20:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:18 UTC)" (scheduled at 2025-02-07 04:20:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:20:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:20:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 8, 732871), True, datetime.datetime(2025, 2, 7, 4, 20, 8, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:20:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:20:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:20:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:20:10,198 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:20:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:20:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:24.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 04:20:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:40 UTC)" (scheduled at 2025-02-07 04:20:10.424250+00:00)
gcworker stdout | 2025-02-07 04:20:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:20:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901710431, None, 1, 0])
gcworker stdout | 2025-02-07 04:20:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:20:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:20:10,875 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:20:11,387 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:20:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:20:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:41.831521+00:00 (in 29.999499 seconds)
autopruneworker stdout | 2025-02-07 04:20:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:41 UTC)" (scheduled at 2025-02-07 04:20:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:20:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898411836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:20:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:20:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:20:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:20:12,652 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:20:13,609 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:20:13,666 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:20:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:20:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:29.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 04:20:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:29 UTC)" (scheduled at 2025-02-07 04:20:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:20:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:20:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:20:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:20:14,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:20:14,235 [238] [DEBUG] [app] Starting request: urn:request:12ccf0d4-88d9-4ef9-8592-bd0d2d0e5f08 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:14,235 [237] [DEBUG] [app] Starting request: urn:request:a43d90b0-5f32-456a-8bc1-ff5ebf4342c0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:20:14,238 [239] [DEBUG] [app] Starting request: urn:request:d6fc7da5-cf7c-414c-ba93-26ae3ca4bb79 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:14,238 [246] [DEBUG] [app] Starting request: urn:request:01a78caf-12ba-469f-be93-515da159542e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:14,238 [239] [DEBUG] [app] Ending request: urn:request:d6fc7da5-cf7c-414c-ba93-26ae3ca4bb79 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6fc7da5-cf7c-414c-ba93-26ae3ca4bb79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:20:14,238 [246] [DEBUG] [app] Ending request: urn:request:01a78caf-12ba-469f-be93-515da159542e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:01a78caf-12ba-469f-be93-515da159542e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:20:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:20:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:14,241 [238] [DEBUG] [app] Starting request: urn:request:1a2257ef-8ef3-4744-a7a2-0c1f74e5b9d2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:20:14,241 [238] [DEBUG] [app] Ending request: urn:request:1a2257ef-8ef3-4744-a7a2-0c1f74e5b9d2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1a2257ef-8ef3-4744-a7a2-0c1f74e5b9d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:20:14,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:14,241 [236] [DEBUG] [app] Starting request: urn:request:ec439297-2c94-412f-b08d-f5b19153408c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:20:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:14,242 [236] [DEBUG] [app] Ending request: urn:request:ec439297-2c94-412f-b08d-f5b19153408c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec439297-2c94-412f-b08d-f5b19153408c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:20:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:20:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:14,253 [238] [DEBUG] [app] Ending request: urn:request:12ccf0d4-88d9-4ef9-8592-bd0d2d0e5f08 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:12ccf0d4-88d9-4ef9-8592-bd0d2d0e5f08', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:20:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:14,253 [237] [DEBUG] [app] Ending request: urn:request:a43d90b0-5f32-456a-8bc1-ff5ebf4342c0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a43d90b0-5f32-456a-8bc1-ff5ebf4342c0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:20:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:20:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:20:14,730 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:15,118 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:20:15,692 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:20:15,766 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:20:15,880 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:20:16,227 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:20:16,607 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:20:16,611 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:20:16,613 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:20:16,616 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:20:16,618 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:20:16,719 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:20:16,783 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:20:16,810 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:20:17,634 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:20:17,637 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:20:17,640 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:20:18,505 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:20:18,613 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:20:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:20:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 04:20:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:28 UTC)" (scheduled at 2025-02-07 04:20:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:20:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:20:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 18, 732870), True, datetime.datetime(2025, 2, 7, 4, 20, 18, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:20:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:20:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:20:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:20:18,778 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:20:18,781 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:20:18,783 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:20:18,788 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:20:18,793 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:20:18,796 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:20:18,798 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:20:18,801 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:20:18,804 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:20:18,979 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:20:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:20:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:03.419110+00:00 (in 39.997695 seconds)
repositorygcworker stdout | 2025-02-07 04:20:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:23 UTC)" (scheduled at 2025-02-07 04:20:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:20:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:20:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:20:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:20:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:40.424250+00:00 (in 16.000678 seconds)
gcworker stdout | 2025-02-07 04:20:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:54 UTC)" (scheduled at 2025-02-07 04:20:24.423055+00:00)
gcworker stdout | 2025-02-07 04:20:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:20:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:20:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:20:28,012 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:20:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:20:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:38.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 04:20:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:38 UTC)" (scheduled at 2025-02-07 04:20:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:20:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:20:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 28, 732905), True, datetime.datetime(2025, 2, 7, 4, 20, 28, 732905), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:20:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:20:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:20:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:20:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:20:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:44.225378+00:00 (in 14.999571 seconds)
quotatotalworker stdout | 2025-02-07 04:20:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:44 UTC)" (scheduled at 2025-02-07 04:20:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:20:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:20:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:20:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:20:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:20:29,236 [235] [DEBUG] [app] Starting request: urn:request:9412c798-4760-4b77-9a8e-e499b2a2a999 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:29,236 [238] [DEBUG] [app] Starting request: urn:request:93d37ca9-8a28-44af-925b-fdb7e95ed827 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:20:29,239 [239] [DEBUG] [app] Starting request: urn:request:d8b7aa4e-ad2a-4bec-8cda-97909a00412c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:29,239 [246] [DEBUG] [app] Starting request: urn:request:0a6e8a9b-63d0-4135-a751-5631fd7ccc98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:29,239 [239] [DEBUG] [app] Ending request: urn:request:d8b7aa4e-ad2a-4bec-8cda-97909a00412c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8b7aa4e-ad2a-4bec-8cda-97909a00412c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:20:29,239 [246] [DEBUG] [app] Ending request: urn:request:0a6e8a9b-63d0-4135-a751-5631fd7ccc98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a6e8a9b-63d0-4135-a751-5631fd7ccc98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:20:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:20:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:29,242 [235] [DEBUG] [app] Starting request: urn:request:98b6cbac-aa4f-4119-9838-615b75cbb556 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:20:29,242 [235] [DEBUG] [app] Ending request: urn:request:98b6cbac-aa4f-4119-9838-615b75cbb556 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98b6cbac-aa4f-4119-9838-615b75cbb556', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:20:29,242 [238] [DEBUG] [app] Starting request: urn:request:8a75b411-3f27-4525-a9c0-c2b72613299d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:20:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:29,243 [238] [DEBUG] [app] Ending request: urn:request:8a75b411-3f27-4525-a9c0-c2b72613299d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8a75b411-3f27-4525-a9c0-c2b72613299d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:20:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:20:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:29,254 [238] [DEBUG] [app] Ending request: urn:request:93d37ca9-8a28-44af-925b-fdb7e95ed827 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93d37ca9-8a28-44af-925b-fdb7e95ed827', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:20:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:20:29,254 [235] [DEBUG] [app] Ending request: urn:request:9412c798-4760-4b77-9a8e-e499b2a2a999 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9412c798-4760-4b77-9a8e-e499b2a2a999', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:20:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:20:29,868 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:20:30,961 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:20:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:20:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:32.625945+00:00 (in 1.004375 seconds)
namespacegcworker stdout | 2025-02-07 04:20:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:31 UTC)" (scheduled at 2025-02-07 04:20:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:20:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:20:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 31, 621870), True, datetime.datetime(2025, 2, 7, 4, 20, 31, 621870), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:20:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:20:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:20:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:20:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:20:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:31.621131+00:00 (in 58.994744 seconds)
namespacegcworker stdout | 2025-02-07 04:20:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:32 UTC)" (scheduled at 2025-02-07 04:20:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:20:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:20:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:20:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:20:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:03.919286+00:00 (in 29.999525 seconds)
buildlogsarchiver stdout | 2025-02-07 04:20:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:03 UTC)" (scheduled at 2025-02-07 04:20:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:20:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 20, 33, 920011), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:20:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:20:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:20:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:20:35,308 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:20:36,388 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:56.713547+00:00 (in 18.997211 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:37 UTC)" (scheduled at 2025-02-07 04:20:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:20:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:20:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:48.732147+00:00 (in 9.999572 seconds)
notificationworker stdout | 2025-02-07 04:20:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:48 UTC)" (scheduled at 2025-02-07 04:20:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:20:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:20:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 38, 732848), True, datetime.datetime(2025, 2, 7, 4, 20, 38, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:20:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:20:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:20:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:20:40,210 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:20:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:20:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:54.423055+00:00 (in 13.998274 seconds)
gcworker stdout | 2025-02-07 04:20:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:10 UTC)" (scheduled at 2025-02-07 04:20:40.424250+00:00)
gcworker stdout | 2025-02-07 04:20:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:20:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901740431, None, 1, 0])
gcworker stdout | 2025-02-07 04:20:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:20:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:20:40,895 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:20:41,401 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:20:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:20:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:11.831521+00:00 (in 29.999528 seconds)
autopruneworker stdout | 2025-02-07 04:20:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:11 UTC)" (scheduled at 2025-02-07 04:20:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:20:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898441836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:20:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:20:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:20:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:20:42,681 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:20:43,645 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:20:43,681 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:20:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:20:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:59.225378+00:00 (in 14.999577 seconds)
quotatotalworker stdout | 2025-02-07 04:20:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:59 UTC)" (scheduled at 2025-02-07 04:20:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:20:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:20:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:20:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:20:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:20:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:20:44,235 [236] [DEBUG] [app] Starting request: urn:request:4aa6e8db-ebf0-4bdf-bc32-53f7ca20245c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:44,235 [235] [DEBUG] [app] Starting request: urn:request:319329d1-0d43-4641-87d2-fca3b1abe640 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:20:44,238 [246] [DEBUG] [app] Starting request: urn:request:86dc1e84-6c25-4bcb-95a2-16988028e6e4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:44,238 [239] [DEBUG] [app] Starting request: urn:request:2eee3b74-d37b-48bf-aa89-66ef394d93e1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:44,239 [246] [DEBUG] [app] Ending request: urn:request:86dc1e84-6c25-4bcb-95a2-16988028e6e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86dc1e84-6c25-4bcb-95a2-16988028e6e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:20:44,239 [239] [DEBUG] [app] Ending request: urn:request:2eee3b74-d37b-48bf-aa89-66ef394d93e1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2eee3b74-d37b-48bf-aa89-66ef394d93e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:20:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:20:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:20:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:44,242 [237] [DEBUG] [app] Starting request: urn:request:fdff3b35-335d-48f8-8f59-8e8e89b8bf08 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:20:44,242 [238] [DEBUG] [app] Starting request: urn:request:6d584746-b754-41d3-8a7f-4bf1f83ffde9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:20:44,242 [237] [DEBUG] [app] Ending request: urn:request:fdff3b35-335d-48f8-8f59-8e8e89b8bf08 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdff3b35-335d-48f8-8f59-8e8e89b8bf08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:20:44,242 [238] [DEBUG] [app] Ending request: urn:request:6d584746-b754-41d3-8a7f-4bf1f83ffde9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d584746-b754-41d3-8a7f-4bf1f83ffde9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:20:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:20:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:44,254 [235] [DEBUG] [app] Ending request: urn:request:319329d1-0d43-4641-87d2-fca3b1abe640 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:319329d1-0d43-4641-87d2-fca3b1abe640', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:20:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:20:44,254 [236] [DEBUG] [app] Ending request: urn:request:4aa6e8db-ebf0-4bdf-bc32-53f7ca20245c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4aa6e8db-ebf0-4bdf-bc32-53f7ca20245c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:20:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
autopruneworker stdout | 2025-02-07 04:20:44,754 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:45,152 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:20:45,707 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:20:45,781 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:20:45,916 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:20:46,248 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:20:46,615 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:20:46,618 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:20:46,621 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:20:46,624 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:20:46,628 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:20:46,733 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:20:46,820 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:20:46,829 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:20:47,641 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:20:47,644 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:20:47,648 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:20:48,541 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:20:48,640 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:20:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:20:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:50.814677+00:00 (in 2.082092 seconds)
notificationworker stdout | 2025-02-07 04:20:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:58 UTC)" (scheduled at 2025-02-07 04:20:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:20:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:20:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 48, 732871), True, datetime.datetime(2025, 2, 7, 4, 20, 48, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:20:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:20:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:20:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:20:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:20:48,788 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:20:48,791 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:20:48,794 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:20:48,798 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:20:48,801 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:20:48,804 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:20:48,806 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:20:48,811 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:20:48,815 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:20:49,016 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:20:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:20:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:20:58.732147+00:00 (in 7.917029 seconds)
notificationworker stdout | 2025-02-07 04:20:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:50 UTC)" (scheduled at 2025-02-07 04:20:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:20:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:20:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:20:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:20:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:10.424250+00:00 (in 16.000716 seconds)
gcworker stdout | 2025-02-07 04:20:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:24 UTC)" (scheduled at 2025-02-07 04:20:54.423055+00:00)
gcworker stdout | 2025-02-07 04:20:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:20:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:20:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:20:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:56.118855+00:00 (in 59.999511 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:20:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:56 UTC)" (scheduled at 2025-02-07 04:20:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:20:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:20:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:20:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:37.715878+00:00 (in 41.001889 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:56 UTC)" (scheduled at 2025-02-07 04:20:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:20:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:20:58,025 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:20:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:20:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:08.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 04:20:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:08 UTC)" (scheduled at 2025-02-07 04:20:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:20:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:20:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 20, 58, 732869), True, datetime.datetime(2025, 2, 7, 4, 20, 58, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:20:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:20:58,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:20:58,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:20:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:20:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:14.225378+00:00 (in 14.999572 seconds)
quotatotalworker stdout | 2025-02-07 04:20:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:14 UTC)" (scheduled at 2025-02-07 04:20:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:20:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:20:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:20:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:20:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:20:59,235 [237] [DEBUG] [app] Starting request: urn:request:eb6db544-301e-4cbd-b845-2b4f526574a0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:59,235 [238] [DEBUG] [app] Starting request: urn:request:213c5f13-e20c-4c59-85f9-2ed194627e62 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:20:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:20:59,238 [246] [DEBUG] [app] Starting request: urn:request:59cd427d-4307-49cd-876e-47221b508ed7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:59,238 [239] [DEBUG] [app] Starting request: urn:request:3e0dd770-30bf-49e3-b3c6-39aac640dbc4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:20:59,238 [246] [DEBUG] [app] Ending request: urn:request:59cd427d-4307-49cd-876e-47221b508ed7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59cd427d-4307-49cd-876e-47221b508ed7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:20:59,238 [239] [DEBUG] [app] Ending request: urn:request:3e0dd770-30bf-49e3-b3c6-39aac640dbc4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e0dd770-30bf-49e3-b3c6-39aac640dbc4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:20:59,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:20:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:20:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:20:59,241 [237] [DEBUG] [app] Starting request: urn:request:9ba166bb-3a48-4a12-beb0-a1d67541fd89 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:20:59,241 [237] [DEBUG] [app] Ending request: urn:request:9ba166bb-3a48-4a12-beb0-a1d67541fd89 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ba166bb-3a48-4a12-beb0-a1d67541fd89', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:20:59,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:20:59,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:59,242 [236] [DEBUG] [app] Starting request: urn:request:40727f08-f8f6-4789-962c-6fc7e2a53ce6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:20:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:59,242 [236] [DEBUG] [app] Ending request: urn:request:40727f08-f8f6-4789-962c-6fc7e2a53ce6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:40727f08-f8f6-4789-962c-6fc7e2a53ce6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:20:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:20:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:20:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:20:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:20:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:20:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:20:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:20:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:20:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:20:59,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:20:59,252 [237] [DEBUG] [app] Ending request: urn:request:eb6db544-301e-4cbd-b845-2b4f526574a0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb6db544-301e-4cbd-b845-2b4f526574a0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:20:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:20:59,254 [238] [DEBUG] [app] Ending request: urn:request:213c5f13-e20c-4c59-85f9-2ed194627e62 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:213c5f13-e20c-4c59-85f9-2ed194627e62', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:20:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:20:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:20:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:20:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:20:59,901 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 04:20:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:20:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:04.936688+00:00 (in 4.998324 seconds)
exportactionlogsworker stdout | 2025-02-07 04:20:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:59 UTC)" (scheduled at 2025-02-07 04:20:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:20:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:20:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:21:59 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:21:00,985 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:21:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:21:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:23.420886+00:00 (in 20.001249 seconds)
repositorygcworker stdout | 2025-02-07 04:21:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:03 UTC)" (scheduled at 2025-02-07 04:21:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:21:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:21:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 3, 419953), True, datetime.datetime(2025, 2, 7, 4, 21, 3, 419953), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:21:03,428 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:21:03,428 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:21:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:21:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:21:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:33.919286+00:00 (in 29.999535 seconds)
buildlogsarchiver stdout | 2025-02-07 04:21:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:33 UTC)" (scheduled at 2025-02-07 04:21:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:21:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 21, 3, 920045), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:21:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:21:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:21:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:21:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:21:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:59.937899+00:00 (in 55.000757 seconds)
exportactionlogsworker stdout | 2025-02-07 04:21:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:04 UTC)" (scheduled at 2025-02-07 04:21:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:21:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:21:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 4, 937480), True, datetime.datetime(2025, 2, 7, 4, 21, 4, 937480), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:21:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:21:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:21:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:21:05,334 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:21:06,423 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:21:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:21:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:18.732147+00:00 (in 9.999447 seconds)
notificationworker stdout | 2025-02-07 04:21:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:18 UTC)" (scheduled at 2025-02-07 04:21:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:21:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:21:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 8, 733143), True, datetime.datetime(2025, 2, 7, 4, 21, 8, 733143), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:21:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:21:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:21:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:21:10,233 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:21:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:21:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:24.423055+00:00 (in 13.998371 seconds)
gcworker stdout | 2025-02-07 04:21:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:40 UTC)" (scheduled at 2025-02-07 04:21:10.424250+00:00)
gcworker stdout | 2025-02-07 04:21:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:21:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901770431, None, 1, 0])
gcworker stdout | 2025-02-07 04:21:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:21:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:21:10,926 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:21:11,418 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:21:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:21:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:41.831521+00:00 (in 29.999553 seconds)
autopruneworker stdout | 2025-02-07 04:21:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:41 UTC)" (scheduled at 2025-02-07 04:21:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:21:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898471836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:21:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:21:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:21:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:21:12,698 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:21:13,664 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:21:13,711 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:21:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:21:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:29.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:21:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:29 UTC)" (scheduled at 2025-02-07 04:21:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:21:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:21:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:21:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:21:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:21:14,235 [235] [DEBUG] [app] Starting request: urn:request:f23e8094-4b94-42d6-9a24-4aaacc0e09bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:14,235 [238] [DEBUG] [app] Starting request: urn:request:d45d25b5-0454-427d-8816-11ca45db8bb1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:21:14,238 [239] [DEBUG] [app] Starting request: urn:request:9a4aa957-6aa4-46fa-8de3-825290d5c01a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:14,238 [245] [DEBUG] [app] Starting request: urn:request:3f78d618-3646-4bfd-8a1d-44867606d4d8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:14,239 [239] [DEBUG] [app] Ending request: urn:request:9a4aa957-6aa4-46fa-8de3-825290d5c01a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a4aa957-6aa4-46fa-8de3-825290d5c01a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:21:14,239 [245] [DEBUG] [app] Ending request: urn:request:3f78d618-3646-4bfd-8a1d-44867606d4d8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f78d618-3646-4bfd-8a1d-44867606d4d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:21:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:21:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:21:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:14,241 [238] [DEBUG] [app] Starting request: urn:request:9445fcb2-79a5-4b6f-be5d-7ca9778409d6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:14,242 [236] [DEBUG] [app] Starting request: urn:request:3af0f41f-1a33-497f-b356-306c3be128a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:14,242 [238] [DEBUG] [app] Ending request: urn:request:9445fcb2-79a5-4b6f-be5d-7ca9778409d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9445fcb2-79a5-4b6f-be5d-7ca9778409d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:21:14,242 [236] [DEBUG] [app] Ending request: urn:request:3af0f41f-1a33-497f-b356-306c3be128a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3af0f41f-1a33-497f-b356-306c3be128a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:21:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:21:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:21:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:21:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:14,253 [238] [DEBUG] [app] Ending request: urn:request:d45d25b5-0454-427d-8816-11ca45db8bb1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d45d25b5-0454-427d-8816-11ca45db8bb1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:14,253 [235] [DEBUG] [app] Ending request: urn:request:f23e8094-4b94-42d6-9a24-4aaacc0e09bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f23e8094-4b94-42d6-9a24-4aaacc0e09bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:21:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:21:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:21:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
autopruneworker stdout | 2025-02-07 04:21:14,790 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:15,188 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:21:15,716 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:21:15,818 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:21:15,932 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:21:16,282 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:21:16,622 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:21:16,626 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:21:16,629 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:21:16,633 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:21:16,636 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:21:16,769 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:21:16,856 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:21:16,865 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:21:17,648 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:21:17,652 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:21:17,655 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:21:18,578 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:21:18,676 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:21:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:21:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:28.732147+00:00 (in 9.999492 seconds)
notificationworker stdout | 2025-02-07 04:21:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:28 UTC)" (scheduled at 2025-02-07 04:21:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:21:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:21:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 18, 732963), True, datetime.datetime(2025, 2, 7, 4, 21, 18, 732963), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:21:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:21:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:21:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:21:18,797 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:21:18,799 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:21:18,804 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:21:18,810 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:21:18,813 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:21:18,816 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:21:18,819 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:21:18,823 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:21:18,825 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:21:19,052 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:21:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:21:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:03.419110+00:00 (in 39.997761 seconds)
repositorygcworker stdout | 2025-02-07 04:21:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:23 UTC)" (scheduled at 2025-02-07 04:21:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:21:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:21:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:21:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:21:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:40.424250+00:00 (in 16.000756 seconds)
gcworker stdout | 2025-02-07 04:21:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:54 UTC)" (scheduled at 2025-02-07 04:21:24.423055+00:00)
gcworker stdout | 2025-02-07 04:21:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:21:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:21:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:21:28,062 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:21:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:21:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:38.732147+00:00 (in 9.999481 seconds)
notificationworker stdout | 2025-02-07 04:21:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:38 UTC)" (scheduled at 2025-02-07 04:21:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:21:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:21:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 28, 732988), True, datetime.datetime(2025, 2, 7, 4, 21, 28, 732988), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:21:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:21:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:21:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:21:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:21:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:44.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 04:21:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:44 UTC)" (scheduled at 2025-02-07 04:21:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:21:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:21:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:21:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:21:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:21:29,235 [235] [DEBUG] [app] Starting request: urn:request:ad7da9be-e365-4931-96be-ad94115dd59d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:29,235 [237] [DEBUG] [app] Starting request: urn:request:7f6a6b50-0845-4dec-8198-7d8e1837b06e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:21:29,238 [239] [DEBUG] [app] Starting request: urn:request:614f4e11-fa50-490c-ab4c-f75dba4a0881 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:29,239 [246] [DEBUG] [app] Starting request: urn:request:8a08cdd8-2654-4710-82ab-7e2048ba6381 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:29,239 [239] [DEBUG] [app] Ending request: urn:request:614f4e11-fa50-490c-ab4c-f75dba4a0881 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:614f4e11-fa50-490c-ab4c-f75dba4a0881', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:21:29,239 [246] [DEBUG] [app] Ending request: urn:request:8a08cdd8-2654-4710-82ab-7e2048ba6381 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a08cdd8-2654-4710-82ab-7e2048ba6381', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:21:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:21:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:29,242 [237] [DEBUG] [app] Starting request: urn:request:c72cb86f-9ad3-4c51-9903-faa64f0b8bce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:29,243 [237] [DEBUG] [app] Ending request: urn:request:c72cb86f-9ad3-4c51-9903-faa64f0b8bce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c72cb86f-9ad3-4c51-9903-faa64f0b8bce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:21:29,243 [235] [DEBUG] [app] Starting request: urn:request:ae815535-f81d-431c-aa57-b87fad229cd2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:21:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:29,243 [235] [DEBUG] [app] Ending request: urn:request:ae815535-f81d-431c-aa57-b87fad229cd2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae815535-f81d-431c-aa57-b87fad229cd2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:21:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:21:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:21:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:29,255 [237] [DEBUG] [app] Ending request: urn:request:7f6a6b50-0845-4dec-8198-7d8e1837b06e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f6a6b50-0845-4dec-8198-7d8e1837b06e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:21:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
gunicorn-web stdout | 2025-02-07 04:21:29,256 [235] [DEBUG] [app] Ending request: urn:request:ad7da9be-e365-4931-96be-ad94115dd59d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ad7da9be-e365-4931-96be-ad94115dd59d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
gunicorn-web stdout | 2025-02-07 04:21:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:21:29,932 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:21:31,021 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:21:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:21:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:32.625945+00:00 (in 1.004342 seconds)
namespacegcworker stdout | 2025-02-07 04:21:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:31 UTC)" (scheduled at 2025-02-07 04:21:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:21:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:21:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 31, 621888), True, datetime.datetime(2025, 2, 7, 4, 21, 31, 621888), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:21:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:21:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:21:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:21:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:21:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:13.624301+00:00 (in 40.997886 seconds)
namespacegcworker stdout | 2025-02-07 04:21:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:32 UTC)" (scheduled at 2025-02-07 04:21:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:21:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:21:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:21:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:21:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:03.919286+00:00 (in 29.999507 seconds)
buildlogsarchiver stdout | 2025-02-07 04:21:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:03 UTC)" (scheduled at 2025-02-07 04:21:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:21:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 21, 33, 920065), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:21:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:21:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:21:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:21:35,341 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:21:36,431 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:56.713547+00:00 (in 18.997206 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:37 UTC)" (scheduled at 2025-02-07 04:21:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:21:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:21:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:48.732147+00:00 (in 9.999506 seconds)
notificationworker stdout | 2025-02-07 04:21:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:48 UTC)" (scheduled at 2025-02-07 04:21:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:21:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:21:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 38, 732920), True, datetime.datetime(2025, 2, 7, 4, 21, 38, 732920), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:21:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:21:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:21:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:21:40,249 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:21:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:21:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:54.423055+00:00 (in 13.998346 seconds)
gcworker stdout | 2025-02-07 04:21:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:10 UTC)" (scheduled at 2025-02-07 04:21:40.424250+00:00)
gcworker stdout | 2025-02-07 04:21:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:21:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901800431, None, 1, 0])
gcworker stdout | 2025-02-07 04:21:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:21:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:21:40,963 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:21:41,450 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:21:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:21:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:11.831521+00:00 (in 29.999495 seconds)
autopruneworker stdout | 2025-02-07 04:21:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:11 UTC)" (scheduled at 2025-02-07 04:21:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:21:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898501836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:21:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:21:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:21:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:21:42,734 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:21:43,673 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:21:43,717 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:21:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:21:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:59.225378+00:00 (in 14.999578 seconds)
quotatotalworker stdout | 2025-02-07 04:21:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:59 UTC)" (scheduled at 2025-02-07 04:21:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:21:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:21:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:21:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:21:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:21:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:21:44,235 [235] [DEBUG] [app] Starting request: urn:request:18963547-b637-4c7f-b31a-f4d3709c03ba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:44,235 [238] [DEBUG] [app] Starting request: urn:request:9c8fed1a-983f-4fe9-a180-b100a9c048e0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:21:44,238 [239] [DEBUG] [app] Starting request: urn:request:076674ad-41d0-4286-b4fd-976461456f01 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:44,238 [246] [DEBUG] [app] Starting request: urn:request:c7d9d547-f7d8-4f2e-bd15-50788037c205 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:44,238 [239] [DEBUG] [app] Ending request: urn:request:076674ad-41d0-4286-b4fd-976461456f01 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:076674ad-41d0-4286-b4fd-976461456f01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:21:44,238 [246] [DEBUG] [app] Ending request: urn:request:c7d9d547-f7d8-4f2e-bd15-50788037c205 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c7d9d547-f7d8-4f2e-bd15-50788037c205', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:21:44,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:21:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:21:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:44,241 [236] [DEBUG] [app] Starting request: urn:request:b234b874-5833-4b30-a0c8-023b0251a3f3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:44,241 [236] [DEBUG] [app] Ending request: urn:request:b234b874-5833-4b30-a0c8-023b0251a3f3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b234b874-5833-4b30-a0c8-023b0251a3f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:21:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:44,242 [237] [DEBUG] [app] Starting request: urn:request:92c0331b-5863-4d05-a6a1-10713900ec25 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:21:44,242 [237] [DEBUG] [app] Ending request: urn:request:92c0331b-5863-4d05-a6a1-10713900ec25 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92c0331b-5863-4d05-a6a1-10713900ec25', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:21:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:21:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:21:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:44,253 [235] [DEBUG] [app] Ending request: urn:request:18963547-b637-4c7f-b31a-f4d3709c03ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:18963547-b637-4c7f-b31a-f4d3709c03ba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:21:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:21:44,254 [238] [DEBUG] [app] Ending request: urn:request:9c8fed1a-983f-4fe9-a180-b100a9c048e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c8fed1a-983f-4fe9-a180-b100a9c048e0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:21:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:21:44,797 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:45,218 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:21:45,734 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:21:45,831 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:21:45,939 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:21:46,291 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:21:46,631 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:21:46,634 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:21:46,638 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:21:46,640 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:21:46,643 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:21:46,806 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:21:46,892 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:21:46,901 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:21:47,657 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:21:47,660 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:21:47,663 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:21:48,614 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:21:48,682 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:21:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:21:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:50.814677+00:00 (in 2.082124 seconds)
notificationworker stdout | 2025-02-07 04:21:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:58 UTC)" (scheduled at 2025-02-07 04:21:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:21:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:21:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 48, 732833), True, datetime.datetime(2025, 2, 7, 4, 21, 48, 732833), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:21:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:21:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:21:48,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:21:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:21:48,805 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:21:48,809 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:21:48,814 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:21:48,820 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:21:48,824 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:21:48,827 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:21:48,830 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:21:48,834 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:21:48,836 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:21:49,079 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:21:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:21:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:21:58.732147+00:00 (in 7.916999 seconds)
notificationworker stdout | 2025-02-07 04:21:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:50 UTC)" (scheduled at 2025-02-07 04:21:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:21:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:21:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:21:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:21:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:10.424250+00:00 (in 16.000731 seconds)
gcworker stdout | 2025-02-07 04:21:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:24 UTC)" (scheduled at 2025-02-07 04:21:54.423055+00:00)
gcworker stdout | 2025-02-07 04:21:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:21:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:21:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:21:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:56.118855+00:00 (in 59.999496 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:21:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:56 UTC)" (scheduled at 2025-02-07 04:21:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:21:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:21:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:21:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:37.715878+00:00 (in 41.001856 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:56 UTC)" (scheduled at 2025-02-07 04:21:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:21:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:21:58,098 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:21:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:21:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:08.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 04:21:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:08 UTC)" (scheduled at 2025-02-07 04:21:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:21:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:21:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 21, 58, 732855), True, datetime.datetime(2025, 2, 7, 4, 21, 58, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:21:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:21:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:21:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:21:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:21:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:14.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 04:21:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:14 UTC)" (scheduled at 2025-02-07 04:21:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:21:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:21:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:21:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:21:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:21:59,235 [237] [DEBUG] [app] Starting request: urn:request:cb190d91-2a54-454e-a244-e1b189d3eeed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:59,235 [235] [DEBUG] [app] Starting request: urn:request:86bce470-0977-4d3b-a9e0-344a6a21660c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:21:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:21:59,238 [239] [DEBUG] [app] Starting request: urn:request:50a3470a-1254-4851-a023-a9660855d102 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:59,238 [245] [DEBUG] [app] Starting request: urn:request:943c5b2c-d998-4a6b-a94b-52a7be848392 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:21:59,239 [245] [DEBUG] [app] Ending request: urn:request:943c5b2c-d998-4a6b-a94b-52a7be848392 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:943c5b2c-d998-4a6b-a94b-52a7be848392', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:21:59,239 [239] [DEBUG] [app] Ending request: urn:request:50a3470a-1254-4851-a023-a9660855d102 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50a3470a-1254-4851-a023-a9660855d102', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:21:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:21:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:21:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:21:59,242 [237] [DEBUG] [app] Starting request: urn:request:4f2df03d-afb2-4383-a3b6-b464b8e7b030 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:59,242 [237] [DEBUG] [app] Ending request: urn:request:4f2df03d-afb2-4383-a3b6-b464b8e7b030 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f2df03d-afb2-4383-a3b6-b464b8e7b030', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:21:59,242 [238] [DEBUG] [app] Starting request: urn:request:97f01063-173e-4871-b174-858ea93ec0ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:21:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:21:59,242 [238] [DEBUG] [app] Ending request: urn:request:97f01063-173e-4871-b174-858ea93ec0ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97f01063-173e-4871-b174-858ea93ec0ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:21:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:21:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:21:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:21:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:21:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:21:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:21:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:21:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:21:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:21:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:21:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:21:59,254 [237] [DEBUG] [app] Ending request: urn:request:cb190d91-2a54-454e-a244-e1b189d3eeed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb190d91-2a54-454e-a244-e1b189d3eeed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:59,254 [235] [DEBUG] [app] Ending request: urn:request:86bce470-0977-4d3b-a9e0-344a6a21660c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:86bce470-0977-4d3b-a9e0-344a6a21660c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:21:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:21:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:21:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:21:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:21:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:21:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 04:21:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:21:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:04.936688+00:00 (in 4.998286 seconds)
exportactionlogsworker stdout | 2025-02-07 04:21:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:59 UTC)" (scheduled at 2025-02-07 04:21:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:21:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:21:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:22:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:21:59,949 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:22:01,056 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:22:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:22:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:23.420886+00:00 (in 20.001324 seconds)
repositorygcworker stdout | 2025-02-07 04:22:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:03 UTC)" (scheduled at 2025-02-07 04:22:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:22:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:22:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 3, 419863), True, datetime.datetime(2025, 2, 7, 4, 22, 3, 419863), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:22:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:22:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:22:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:22:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:22:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:33.919286+00:00 (in 29.999510 seconds)
buildlogsarchiver stdout | 2025-02-07 04:22:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:33 UTC)" (scheduled at 2025-02-07 04:22:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:22:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 22, 3, 920029), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:22:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:22:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:22:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:22:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:22:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:59.937899+00:00 (in 55.000787 seconds)
exportactionlogsworker stdout | 2025-02-07 04:22:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:04 UTC)" (scheduled at 2025-02-07 04:22:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:22:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:22:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 4, 937448), True, datetime.datetime(2025, 2, 7, 4, 22, 4, 937448), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:22:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:22:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:22:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:22:05,361 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:22:06,467 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:22:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:18.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 04:22:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:18 UTC)" (scheduled at 2025-02-07 04:22:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:22:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:22:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 8, 732857), True, datetime.datetime(2025, 2, 7, 4, 22, 8, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:22:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:22:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:22:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:22:10,285 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:22:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:22:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:24.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 04:22:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:40 UTC)" (scheduled at 2025-02-07 04:22:10.424250+00:00)
gcworker stdout | 2025-02-07 04:22:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:22:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901830431, None, 1, 0])
gcworker stdout | 2025-02-07 04:22:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:22:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:22:10,999 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:22:11,480 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:22:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:22:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:41.831521+00:00 (in 29.999478 seconds)
autopruneworker stdout | 2025-02-07 04:22:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:41 UTC)" (scheduled at 2025-02-07 04:22:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:22:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898531836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:22:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:22:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:22:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:22:12,769 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
namespacegcworker stdout | 2025-02-07 04:22:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:22:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:31.621131+00:00 (in 17.996389 seconds)
namespacegcworker stdout | 2025-02-07 04:22:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:27:13 UTC)" (scheduled at 2025-02-07 04:22:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:22:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 22, 13, 624983), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:22:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 22, 13, 624983), True, datetime.datetime(2025, 2, 7, 4, 22, 13, 624983), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:22:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 22, 13, 624983), True, datetime.datetime(2025, 2, 7, 4, 22, 13, 624983), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 22, 13, 624983), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:22:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:22:13,634 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:27:13 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:22:13,710 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:22:13,749 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:22:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:22:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:29.225378+00:00 (in 14.999477 seconds)
quotatotalworker stdout | 2025-02-07 04:22:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:29 UTC)" (scheduled at 2025-02-07 04:22:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:22:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:22:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:22:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:22:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:22:14,235 [237] [DEBUG] [app] Starting request: urn:request:61ff7ca8-62cc-4ece-9b17-31626d504985 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:14,235 [236] [DEBUG] [app] Starting request: urn:request:10bc2884-d69d-491b-8180-eb650cbac85c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:22:14,238 [244] [DEBUG] [app] Starting request: urn:request:1d0994ab-bf44-4502-a59d-b270fa11985d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:14,238 [246] [DEBUG] [app] Starting request: urn:request:206f1ea3-11c2-43ab-b291-13ebbe3555ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:14,238 [246] [DEBUG] [app] Ending request: urn:request:206f1ea3-11c2-43ab-b291-13ebbe3555ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:206f1ea3-11c2-43ab-b291-13ebbe3555ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:22:14,238 [244] [DEBUG] [app] Ending request: urn:request:1d0994ab-bf44-4502-a59d-b270fa11985d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1d0994ab-bf44-4502-a59d-b270fa11985d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:22:14,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:22:14,238 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:22:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:14,241 [236] [DEBUG] [app] Starting request: urn:request:19fca001-f3a7-4c68-8388-5c5d2238d06d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:22:14,241 [238] [DEBUG] [app] Starting request: urn:request:cdfcf124-8c03-4318-b3b5-9066272c700c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:22:14,241 [236] [DEBUG] [app] Ending request: urn:request:19fca001-f3a7-4c68-8388-5c5d2238d06d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:19fca001-f3a7-4c68-8388-5c5d2238d06d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:22:14,241 [238] [DEBUG] [app] Ending request: urn:request:cdfcf124-8c03-4318-b3b5-9066272c700c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cdfcf124-8c03-4318-b3b5-9066272c700c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:22:14,241 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:22:14,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:22:14,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:22:14,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:14,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:22:14,245 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:14,245 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:14,245 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:14,245 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:14,249 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:14,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:14,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:14,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:14,252 [237] [DEBUG] [app] Ending request: urn:request:61ff7ca8-62cc-4ece-9b17-31626d504985 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61ff7ca8-62cc-4ece-9b17-31626d504985', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:14,252 [236] [DEBUG] [app] Ending request: urn:request:10bc2884-d69d-491b-8180-eb650cbac85c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10bc2884-d69d-491b-8180-eb650cbac85c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:14,252 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:22:14,252 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:22:14,252 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:22:14,252 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
autopruneworker stdout | 2025-02-07 04:22:14,834 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:15,255 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:22:15,768 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:22:15,868 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:22:15,975 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:22:16,314 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:22:16,638 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:22:16,642 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:22:16,645 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:22:16,649 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:22:16,652 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:22:16,825 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:22:16,917 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:22:16,938 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:22:17,665 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:22:17,668 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:22:17,671 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:22:18,624 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:22:18,714 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:22:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:28.732147+00:00 (in 9.999533 seconds)
notificationworker stdout | 2025-02-07 04:22:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:28 UTC)" (scheduled at 2025-02-07 04:22:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:22:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:22:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 18, 732920), True, datetime.datetime(2025, 2, 7, 4, 22, 18, 732920), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:22:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:22:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:22:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:22:18,819 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:22:18,822 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:22:18,824 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:22:18,829 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:22:18,832 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:22:18,837 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:22:18,841 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:22:18,844 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:22:18,847 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:22:19,115 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:22:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:22:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:03.419110+00:00 (in 39.997760 seconds)
repositorygcworker stdout | 2025-02-07 04:22:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:23 UTC)" (scheduled at 2025-02-07 04:22:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:22:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:22:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:22:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:22:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:40.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 04:22:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:54 UTC)" (scheduled at 2025-02-07 04:22:24.423055+00:00)
gcworker stdout | 2025-02-07 04:22:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:22:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:22:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:22:28,134 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:22:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:33.814166+00:00 (in 5.081505 seconds)
notificationworker stdout | 2025-02-07 04:22:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:38 UTC)" (scheduled at 2025-02-07 04:22:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:22:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:22:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 28, 732954), True, datetime.datetime(2025, 2, 7, 4, 22, 28, 732954), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:22:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:22:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:22:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:22:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:22:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:44.225378+00:00 (in 14.999478 seconds)
quotatotalworker stdout | 2025-02-07 04:22:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:44 UTC)" (scheduled at 2025-02-07 04:22:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:22:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:22:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:22:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:22:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:22:29,236 [235] [DEBUG] [app] Starting request: urn:request:15ee4647-377f-4879-9c53-73450c121ebb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:29,236 [236] [DEBUG] [app] Starting request: urn:request:3048725f-d2f6-4185-80eb-0ad9748b7a2c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:22:29,239 [242] [DEBUG] [app] Starting request: urn:request:cdf598a9-c656-4b20-95e5-8105c84b1b59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:29,239 [245] [DEBUG] [app] Starting request: urn:request:6eebf1fc-dd11-4893-864c-9b8aac202774 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:29,239 [245] [DEBUG] [app] Ending request: urn:request:6eebf1fc-dd11-4893-864c-9b8aac202774 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6eebf1fc-dd11-4893-864c-9b8aac202774', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:22:29,239 [242] [DEBUG] [app] Ending request: urn:request:cdf598a9-c656-4b20-95e5-8105c84b1b59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cdf598a9-c656-4b20-95e5-8105c84b1b59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:22:29,239 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:22:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:22:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:29,242 [235] [DEBUG] [app] Starting request: urn:request:ba0c51e2-6789-42aa-b8f3-784bfc290827 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:22:29,242 [236] [DEBUG] [app] Starting request: urn:request:33152a37-f7e8-434c-a403-67a14b34e1ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:22:29,242 [235] [DEBUG] [app] Ending request: urn:request:ba0c51e2-6789-42aa-b8f3-784bfc290827 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba0c51e2-6789-42aa-b8f3-784bfc290827', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:22:29,242 [236] [DEBUG] [app] Ending request: urn:request:33152a37-f7e8-434c-a403-67a14b34e1ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:33152a37-f7e8-434c-a403-67a14b34e1ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:22:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:22:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:22:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:22:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:22:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:29,254 [235] [DEBUG] [app] Ending request: urn:request:15ee4647-377f-4879-9c53-73450c121ebb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15ee4647-377f-4879-9c53-73450c121ebb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:22:29,254 [236] [DEBUG] [app] Ending request: urn:request:3048725f-d2f6-4185-80eb-0ad9748b7a2c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3048725f-d2f6-4185-80eb-0ad9748b7a2c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:22:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:22:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:22:29,958 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:22:31,091 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:22:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:22:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:32.625945+00:00 (in 1.004373 seconds)
namespacegcworker stdout | 2025-02-07 04:22:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:31 UTC)" (scheduled at 2025-02-07 04:22:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:22:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:22:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 31, 621847), True, datetime.datetime(2025, 2, 7, 4, 22, 31, 621847), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:22:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:22:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:22:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:22:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:22:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:31.621131+00:00 (in 58.994735 seconds)
namespacegcworker stdout | 2025-02-07 04:22:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:32 UTC)" (scheduled at 2025-02-07 04:22:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:22:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:22:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:22:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:38.732147+00:00 (in 4.917534 seconds)
notificationworker stdout | 2025-02-07 04:22:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:27:33 UTC)" (scheduled at 2025-02-07 04:22:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:22:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 22, 33, 814857), 'notification/%'])
notificationworker stdout | 2025-02-07 04:22:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 22, 33, 814857), True, datetime.datetime(2025, 2, 7, 4, 22, 33, 814857), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:22:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 22, 33, 814857), True, datetime.datetime(2025, 2, 7, 4, 22, 33, 814857), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 22, 33, 814857), 'notification/%'])
notificationworker stdout | 2025-02-07 04:22:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:22:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:27:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:22:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:22:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:03.919286+00:00 (in 29.999528 seconds)
buildlogsarchiver stdout | 2025-02-07 04:22:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:03 UTC)" (scheduled at 2025-02-07 04:22:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:22:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 22, 33, 920010), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:22:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:22:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:22:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:22:35,397 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:22:36,498 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:56.713547+00:00 (in 18.997205 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:37 UTC)" (scheduled at 2025-02-07 04:22:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:22:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:48.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:22:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:48 UTC)" (scheduled at 2025-02-07 04:22:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:22:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:22:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 38, 732859), True, datetime.datetime(2025, 2, 7, 4, 22, 38, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:22:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:22:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:22:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:22:40,315 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:22:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:22:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:54.423055+00:00 (in 13.998357 seconds)
gcworker stdout | 2025-02-07 04:22:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:10 UTC)" (scheduled at 2025-02-07 04:22:40.424250+00:00)
gcworker stdout | 2025-02-07 04:22:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:22:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901860430, None, 1, 0])
gcworker stdout | 2025-02-07 04:22:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:22:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:22:41,019 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:22:41,497 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:22:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:22:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:11.831521+00:00 (in 29.999459 seconds)
autopruneworker stdout | 2025-02-07 04:22:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:11 UTC)" (scheduled at 2025-02-07 04:22:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:22:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898561836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:22:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:22:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:22:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:22:42,778 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:22:43,723 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:22:43,785 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:22:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:22:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:59.225378+00:00 (in 14.999479 seconds)
quotatotalworker stdout | 2025-02-07 04:22:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:59 UTC)" (scheduled at 2025-02-07 04:22:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:22:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:22:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:22:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:22:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:22:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:22:44,235 [237] [DEBUG] [app] Starting request: urn:request:3f68c1eb-282d-453b-b999-1a058c215d0a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:44,235 [236] [DEBUG] [app] Starting request: urn:request:c30812fd-f9c4-4343-969d-87b9021b38e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:22:44,238 [239] [DEBUG] [app] Starting request: urn:request:3e7840a4-5003-4976-ae02-fbed8fa05a5d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:44,238 [244] [DEBUG] [app] Starting request: urn:request:4d9584c7-79f4-486a-a12e-16dcdf6e205c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:44,238 [244] [DEBUG] [app] Ending request: urn:request:4d9584c7-79f4-486a-a12e-16dcdf6e205c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d9584c7-79f4-486a-a12e-16dcdf6e205c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:22:44,238 [239] [DEBUG] [app] Ending request: urn:request:3e7840a4-5003-4976-ae02-fbed8fa05a5d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e7840a4-5003-4976-ae02-fbed8fa05a5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:22:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:22:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:22:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:22:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:44,241 [237] [DEBUG] [app] Starting request: urn:request:323806c6-d23d-4d6a-b1eb-e7a51262b659 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:22:44,241 [238] [DEBUG] [app] Starting request: urn:request:baa83330-f008-48e2-9b6e-66f38f56938a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:22:44,241 [237] [DEBUG] [app] Ending request: urn:request:323806c6-d23d-4d6a-b1eb-e7a51262b659 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:323806c6-d23d-4d6a-b1eb-e7a51262b659', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:22:44,241 [238] [DEBUG] [app] Ending request: urn:request:baa83330-f008-48e2-9b6e-66f38f56938a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:baa83330-f008-48e2-9b6e-66f38f56938a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:22:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:22:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:22:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:22:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:44,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:22:44,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:44,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:22:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:44,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:44,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:44,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:44,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:44,253 [237] [DEBUG] [app] Ending request: urn:request:3f68c1eb-282d-453b-b999-1a058c215d0a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f68c1eb-282d-453b-b999-1a058c215d0a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:22:44,253 [236] [DEBUG] [app] Ending request: urn:request:c30812fd-f9c4-4343-969d-87b9021b38e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c30812fd-f9c4-4343-969d-87b9021b38e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:22:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:22:44,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
autopruneworker stdout | 2025-02-07 04:22:44,841 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:45,291 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:22:45,799 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:22:45,899 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:22:45,986 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:22:46,350 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:22:46,646 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:22:46,650 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:22:46,652 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:22:46,655 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:22:46,658 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:22:46,858 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:22:46,949 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:22:46,973 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:22:47,673 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:22:47,676 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:22:47,679 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:22:48,660 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
notificationworker stdout | 2025-02-07 04:22:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:50.814677+00:00 (in 2.082138 seconds)
notificationworker stdout | 2025-02-07 04:22:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:58 UTC)" (scheduled at 2025-02-07 04:22:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:22:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:22:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 48, 732818), True, datetime.datetime(2025, 2, 7, 4, 22, 48, 732818), 0, 'notification/%', 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:22:48,735 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:22:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:22:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:22:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:22:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:22:48,831 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:22:48,835 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:22:48,839 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:22:48,842 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:22:48,845 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:22:48,847 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:22:48,852 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:22:48,855 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:22:48,858 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:22:49,142 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:22:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:22:58.732147+00:00 (in 7.917031 seconds)
notificationworker stdout | 2025-02-07 04:22:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:50 UTC)" (scheduled at 2025-02-07 04:22:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:22:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:22:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:22:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:22:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:10.424250+00:00 (in 16.000678 seconds)
gcworker stdout | 2025-02-07 04:22:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:24 UTC)" (scheduled at 2025-02-07 04:22:54.423055+00:00)
gcworker stdout | 2025-02-07 04:22:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:22:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:22:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:22:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:56.118855+00:00 (in 59.999492 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:22:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:56 UTC)" (scheduled at 2025-02-07 04:22:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:22:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:22:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:22:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:37.715878+00:00 (in 41.001860 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:56 UTC)" (scheduled at 2025-02-07 04:22:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:22:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:22:58,170 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:22:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:22:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:08.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 04:22:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:08 UTC)" (scheduled at 2025-02-07 04:22:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:22:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:22:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 22, 58, 732882), True, datetime.datetime(2025, 2, 7, 4, 22, 58, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:22:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:22:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:22:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:22:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:22:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:14.225378+00:00 (in 14.999510 seconds)
quotatotalworker stdout | 2025-02-07 04:22:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:14 UTC)" (scheduled at 2025-02-07 04:22:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:22:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:22:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:22:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:22:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:22:59,235 [235] [DEBUG] [app] Starting request: urn:request:566bd379-3519-4e52-8b72-8aa9b507721e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:59,235 [236] [DEBUG] [app] Starting request: urn:request:7ffd6482-bb40-4226-8da1-4d8a09a0ef98 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:22:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:22:59,238 [239] [DEBUG] [app] Starting request: urn:request:dfa4363c-e938-4bf0-b143-24a1a552ce04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:59,238 [246] [DEBUG] [app] Starting request: urn:request:00f41595-3a9f-40e8-8729-56496aba0a38 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:22:59,238 [239] [DEBUG] [app] Ending request: urn:request:dfa4363c-e938-4bf0-b143-24a1a552ce04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dfa4363c-e938-4bf0-b143-24a1a552ce04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:22:59,238 [246] [DEBUG] [app] Ending request: urn:request:00f41595-3a9f-40e8-8729-56496aba0a38 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:00f41595-3a9f-40e8-8729-56496aba0a38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:22:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:22:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:22:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:22:59,241 [237] [DEBUG] [app] Starting request: urn:request:6833a72b-e637-4ecb-8358-ff122d30aa18 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:22:59,242 [237] [DEBUG] [app] Ending request: urn:request:6833a72b-e637-4ecb-8358-ff122d30aa18 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6833a72b-e637-4ecb-8358-ff122d30aa18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:22:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:22:59,242 [238] [DEBUG] [app] Starting request: urn:request:f04cb751-726b-485c-b968-b93b07a5e863 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:22:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:59,242 [238] [DEBUG] [app] Ending request: urn:request:f04cb751-726b-485c-b968-b93b07a5e863 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f04cb751-726b-485c-b968-b93b07a5e863', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:22:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:22:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:22:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:22:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:22:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:22:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:22:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:22:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:22:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:22:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:22:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:22:59,253 [235] [DEBUG] [app] Ending request: urn:request:566bd379-3519-4e52-8b72-8aa9b507721e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:566bd379-3519-4e52-8b72-8aa9b507721e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:59,253 [236] [DEBUG] [app] Ending request: urn:request:7ffd6482-bb40-4226-8da1-4d8a09a0ef98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7ffd6482-bb40-4226-8da1-4d8a09a0ef98', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:22:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:22:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:22:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:22:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:22:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:22:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:22:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:22:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:04.936688+00:00 (in 4.998337 seconds)
exportactionlogsworker stdout | 2025-02-07 04:22:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:59 UTC)" (scheduled at 2025-02-07 04:22:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:22:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:22:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:23:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:22:59,968 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:23:01,107 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:23:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:23:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:23.420886+00:00 (in 20.001293 seconds)
repositorygcworker stdout | 2025-02-07 04:23:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:03 UTC)" (scheduled at 2025-02-07 04:23:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:23:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:23:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 3, 419900), True, datetime.datetime(2025, 2, 7, 4, 23, 3, 419900), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:23:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:23:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:23:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:23:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:23:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:33.919286+00:00 (in 29.999545 seconds)
buildlogsarchiver stdout | 2025-02-07 04:23:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:33 UTC)" (scheduled at 2025-02-07 04:23:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:23:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 23, 3, 919990), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:23:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:23:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:23:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:23:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:23:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:59.937899+00:00 (in 55.000796 seconds)
exportactionlogsworker stdout | 2025-02-07 04:23:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:04 UTC)" (scheduled at 2025-02-07 04:23:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:23:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:23:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 4, 937395), True, datetime.datetime(2025, 2, 7, 4, 23, 4, 937395), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:23:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:23:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:23:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:23:05,417 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:23:06,534 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:23:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:23:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:18.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:23:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:18 UTC)" (scheduled at 2025-02-07 04:23:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:23:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:23:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 8, 732891), True, datetime.datetime(2025, 2, 7, 4, 23, 8, 732891), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:23:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:23:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:23:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:23:10,352 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:23:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:23:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:24.423055+00:00 (in 13.998282 seconds)
gcworker stdout | 2025-02-07 04:23:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:40 UTC)" (scheduled at 2025-02-07 04:23:10.424250+00:00)
gcworker stdout | 2025-02-07 04:23:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:23:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901890431, None, 1, 0])
gcworker stdout | 2025-02-07 04:23:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:23:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:23:11,055 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:23:11,506 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:23:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:23:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:41.831521+00:00 (in 29.999487 seconds)
autopruneworker stdout | 2025-02-07 04:23:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:41 UTC)" (scheduled at 2025-02-07 04:23:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:23:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898591836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:23:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:23:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:23:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:23:12,810 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:23:13,746 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:23:13,821 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:23:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:23:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:29.225378+00:00 (in 14.999517 seconds)
quotatotalworker stdout | 2025-02-07 04:23:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:29 UTC)" (scheduled at 2025-02-07 04:23:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:23:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:23:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:23:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:23:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:23:14,235 [235] [DEBUG] [app] Starting request: urn:request:b9054f71-73ae-4d71-9278-a4224348504a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:14,235 [238] [DEBUG] [app] Starting request: urn:request:f0f4ce6a-e332-4620-8daa-914851ee831e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:23:14,238 [245] [DEBUG] [app] Starting request: urn:request:72cd7f30-e59f-4a8f-8a7e-93a015798657 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:14,238 [239] [DEBUG] [app] Starting request: urn:request:af3d7a12-90ca-4b18-b70e-8806a8f80a2f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:14,238 [245] [DEBUG] [app] Ending request: urn:request:72cd7f30-e59f-4a8f-8a7e-93a015798657 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:72cd7f30-e59f-4a8f-8a7e-93a015798657', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:23:14,238 [239] [DEBUG] [app] Ending request: urn:request:af3d7a12-90ca-4b18-b70e-8806a8f80a2f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af3d7a12-90ca-4b18-b70e-8806a8f80a2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:23:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:23:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:23:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:14,242 [236] [DEBUG] [app] Starting request: urn:request:1d7f04b3-b5bc-40c9-a78e-8e7e91ac1e78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:14,242 [237] [DEBUG] [app] Starting request: urn:request:d560c91a-36ba-4d5a-bc5d-478df07820f4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:14,242 [236] [DEBUG] [app] Ending request: urn:request:1d7f04b3-b5bc-40c9-a78e-8e7e91ac1e78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d7f04b3-b5bc-40c9-a78e-8e7e91ac1e78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:23:14,242 [237] [DEBUG] [app] Ending request: urn:request:d560c91a-36ba-4d5a-bc5d-478df07820f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d560c91a-36ba-4d5a-bc5d-478df07820f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:23:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:23:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:23:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:14,254 [235] [DEBUG] [app] Ending request: urn:request:b9054f71-73ae-4d71-9278-a4224348504a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b9054f71-73ae-4d71-9278-a4224348504a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:14,254 [238] [DEBUG] [app] Ending request: urn:request:f0f4ce6a-e332-4620-8daa-914851ee831e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0f4ce6a-e332-4620-8daa-914851ee831e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:23:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:23:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:23:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
autopruneworker stdout | 2025-02-07 04:23:14,849 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:15,327 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:23:15,834 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:23:15,907 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:23:16,022 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:23:16,386 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:23:16,653 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:23:16,657 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:23:16,660 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:23:16,663 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:23:16,666 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:23:16,874 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:23:16,984 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:23:17,009 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:23:17,681 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:23:17,684 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:23:17,686 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
namespacegcworker stdout | 2025-02-07 04:23:18,696 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
notificationworker stdout | 2025-02-07 04:23:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:23:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:28.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 04:23:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:28 UTC)" (scheduled at 2025-02-07 04:23:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:23:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:23:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 18, 732876), True, datetime.datetime(2025, 2, 7, 4, 23, 18, 732876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:23:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:23:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:23:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:28 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:23:18,771 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:23:18,843 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:23:18,846 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:23:18,849 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:23:18,852 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:23:18,855 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:23:18,859 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:23:18,861 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:23:18,864 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:23:18,866 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:23:19,178 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:23:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:23:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:32.420707+00:00 (in 8.999326 seconds)
repositorygcworker stdout | 2025-02-07 04:23:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:23 UTC)" (scheduled at 2025-02-07 04:23:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:23:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:23:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:23:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:23:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:40.424250+00:00 (in 16.000707 seconds)
gcworker stdout | 2025-02-07 04:23:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:54 UTC)" (scheduled at 2025-02-07 04:23:24.423055+00:00)
gcworker stdout | 2025-02-07 04:23:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:23:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:23:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:23:28,193 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:23:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:23:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:38.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 04:23:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:38 UTC)" (scheduled at 2025-02-07 04:23:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:23:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:23:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 28, 732905), True, datetime.datetime(2025, 2, 7, 4, 23, 28, 732905), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:23:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:23:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:23:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:23:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:23:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:44.225378+00:00 (in 14.999504 seconds)
quotatotalworker stdout | 2025-02-07 04:23:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:44 UTC)" (scheduled at 2025-02-07 04:23:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:23:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:23:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:23:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:23:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:23:29,235 [235] [DEBUG] [app] Starting request: urn:request:d448df60-4c1e-4372-a89e-1ed1cbf4f577 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:29,235 [238] [DEBUG] [app] Starting request: urn:request:84ecbd35-39c5-4517-ae06-e5078767f124 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:23:29,238 [239] [DEBUG] [app] Starting request: urn:request:88d39b83-d31f-4eb7-8162-ec83044735ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:29,238 [246] [DEBUG] [app] Starting request: urn:request:6581cf6e-0319-4b21-9205-ff66dda13c91 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:29,238 [239] [DEBUG] [app] Ending request: urn:request:88d39b83-d31f-4eb7-8162-ec83044735ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:88d39b83-d31f-4eb7-8162-ec83044735ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:23:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:23:29,239 [246] [DEBUG] [app] Ending request: urn:request:6581cf6e-0319-4b21-9205-ff66dda13c91 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6581cf6e-0319-4b21-9205-ff66dda13c91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:23:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:23:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:23:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:23:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:29,242 [237] [DEBUG] [app] Starting request: urn:request:01ba51b1-6ada-4eb7-892d-065f1d03cced (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:29,242 [235] [DEBUG] [app] Starting request: urn:request:3c701e59-aa00-455b-8e6f-0e43cbaee94e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:29,242 [235] [DEBUG] [app] Ending request: urn:request:3c701e59-aa00-455b-8e6f-0e43cbaee94e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c701e59-aa00-455b-8e6f-0e43cbaee94e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:23:29,242 [237] [DEBUG] [app] Ending request: urn:request:01ba51b1-6ada-4eb7-892d-065f1d03cced (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01ba51b1-6ada-4eb7-892d-065f1d03cced', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:23:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:23:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:23:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:23:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:29,255 [235] [DEBUG] [app] Ending request: urn:request:d448df60-4c1e-4372-a89e-1ed1cbf4f577 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d448df60-4c1e-4372-a89e-1ed1cbf4f577', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:23:29,256 [238] [DEBUG] [app] Ending request: urn:request:84ecbd35-39c5-4517-ae06-e5078767f124 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84ecbd35-39c5-4517-ae06-e5078767f124', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:23:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 04:23:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
repositoryactioncounter stdout | 2025-02-07 04:23:30,004 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:23:31,140 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:23:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:23:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:32.625945+00:00 (in 1.004368 seconds)
namespacegcworker stdout | 2025-02-07 04:23:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:31 UTC)" (scheduled at 2025-02-07 04:23:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:23:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:23:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 31, 621855), True, datetime.datetime(2025, 2, 7, 4, 23, 31, 621855), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:23:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:23:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:23:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:23:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:23:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:03.419110+00:00 (in 30.997976 seconds)
repositorygcworker stdout | 2025-02-07 04:23:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:28:32 UTC)" (scheduled at 2025-02-07 04:23:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:23:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 23, 32, 421392), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:23:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 23, 32, 421392), True, datetime.datetime(2025, 2, 7, 4, 23, 32, 421392), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:23:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 23, 32, 421392), True, datetime.datetime(2025, 2, 7, 4, 23, 32, 421392), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 23, 32, 421392), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:23:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:23:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:28:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:23:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:23:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:31.621131+00:00 (in 58.994727 seconds)
namespacegcworker stdout | 2025-02-07 04:23:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:32 UTC)" (scheduled at 2025-02-07 04:23:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:23:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:23:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:23:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:23:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:03.919286+00:00 (in 29.999553 seconds)
buildlogsarchiver stdout | 2025-02-07 04:23:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:03 UTC)" (scheduled at 2025-02-07 04:23:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:23:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 23, 33, 919974), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:23:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:23:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:23:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:23:35,448 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:23:36,570 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:56.713547+00:00 (in 18.997173 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:37 UTC)" (scheduled at 2025-02-07 04:23:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:23:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:23:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:48.732147+00:00 (in 9.999587 seconds)
notificationworker stdout | 2025-02-07 04:23:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:48 UTC)" (scheduled at 2025-02-07 04:23:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:23:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:23:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 38, 732824), True, datetime.datetime(2025, 2, 7, 4, 23, 38, 732824), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:23:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:23:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:23:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:48 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:23:40,388 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:23:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:23:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:54.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 04:23:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:10 UTC)" (scheduled at 2025-02-07 04:23:40.424250+00:00)
gcworker stdout | 2025-02-07 04:23:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:23:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901920431, None, 1, 0])
gcworker stdout | 2025-02-07 04:23:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:23:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:10 UTC)" executed successfully
builder stdout | 2025-02-07 04:23:41,087 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:23:41,542 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:23:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:23:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:11.831521+00:00 (in 29.999510 seconds)
autopruneworker stdout | 2025-02-07 04:23:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:11 UTC)" (scheduled at 2025-02-07 04:23:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:23:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898621836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:23:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:23:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:23:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:23:42,843 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:23:43,774 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:23:43,854 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:23:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:23:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:59.225378+00:00 (in 14.999532 seconds)
quotatotalworker stdout | 2025-02-07 04:23:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:59 UTC)" (scheduled at 2025-02-07 04:23:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:23:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:23:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:23:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:23:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:23:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:23:44,235 [235] [DEBUG] [app] Starting request: urn:request:ca788123-09bb-4533-8d56-c99cbfcaed85 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:44,235 [238] [DEBUG] [app] Starting request: urn:request:9a66870f-c5cf-4287-8703-ef602da72e9c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:23:44,238 [239] [DEBUG] [app] Starting request: urn:request:4f026c5b-80ab-4c41-aec0-12e6695b4c55 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:44,238 [244] [DEBUG] [app] Starting request: urn:request:26a87732-0891-4378-b305-459008e19a99 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:44,238 [239] [DEBUG] [app] Ending request: urn:request:4f026c5b-80ab-4c41-aec0-12e6695b4c55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f026c5b-80ab-4c41-aec0-12e6695b4c55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:23:44,238 [244] [DEBUG] [app] Ending request: urn:request:26a87732-0891-4378-b305-459008e19a99 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:26a87732-0891-4378-b305-459008e19a99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:23:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:23:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:23:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:44,241 [235] [DEBUG] [app] Starting request: urn:request:21b2943c-2a64-47b7-a6b6-5d3afb538f46 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:44,241 [235] [DEBUG] [app] Ending request: urn:request:21b2943c-2a64-47b7-a6b6-5d3afb538f46 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21b2943c-2a64-47b7-a6b6-5d3afb538f46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:23:44,241 [237] [DEBUG] [app] Starting request: urn:request:b6a4000f-b87f-4cf1-b8d9-ec0bb3721bda (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:23:44,242 [237] [DEBUG] [app] Ending request: urn:request:b6a4000f-b87f-4cf1-b8d9-ec0bb3721bda (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6a4000f-b87f-4cf1-b8d9-ec0bb3721bda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:23:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:23:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:23:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:44,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:44,252 [235] [DEBUG] [app] Ending request: urn:request:ca788123-09bb-4533-8d56-c99cbfcaed85 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca788123-09bb-4533-8d56-c99cbfcaed85', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:44,253 [238] [DEBUG] [app] Ending request: urn:request:9a66870f-c5cf-4287-8703-ef602da72e9c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a66870f-c5cf-4287-8703-ef602da72e9c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:23:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:23:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:23:44,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
autopruneworker stdout | 2025-02-07 04:23:44,859 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:45,363 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:23:45,870 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:23:45,943 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:23:46,042 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:23:46,422 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:23:46,661 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:23:46,666 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:23:46,669 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:23:46,672 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:23:46,675 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:23:46,907 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:23:47,001 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:23:47,027 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:23:47,688 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:23:47,692 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:23:47,695 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:23:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:23:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:50.814677+00:00 (in 2.082113 seconds)
notificationworker stdout | 2025-02-07 04:23:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:58 UTC)" (scheduled at 2025-02-07 04:23:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:23:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:23:48,733 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
notificationworker stdout | 2025-02-07 04:23:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 48, 732848), True, datetime.datetime(2025, 2, 7, 4, 23, 48, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:23:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:23:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:23:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:23:58 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:23:48,808 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:23:48,853 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:23:48,856 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:23:48,859 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:23:48,862 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:23:48,866 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:23:48,869 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:23:48,872 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:23:48,875 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:23:48,877 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:23:49,211 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:23:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:23:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:23:58.732147+00:00 (in 7.916984 seconds)
notificationworker stdout | 2025-02-07 04:23:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:50 UTC)" (scheduled at 2025-02-07 04:23:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:23:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:23:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:23:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:23:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:10.424250+00:00 (in 16.000733 seconds)
gcworker stdout | 2025-02-07 04:23:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:24 UTC)" (scheduled at 2025-02-07 04:23:54.423055+00:00)
gcworker stdout | 2025-02-07 04:23:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:23:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:23:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:23:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:56.118855+00:00 (in 59.999475 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:23:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:56 UTC)" (scheduled at 2025-02-07 04:23:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:23:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:23:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:23:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:37.715878+00:00 (in 41.001861 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:56 UTC)" (scheduled at 2025-02-07 04:23:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:23:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:23:58,206 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:23:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:23:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:08.732147+00:00 (in 9.999473 seconds)
notificationworker stdout | 2025-02-07 04:23:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:08 UTC)" (scheduled at 2025-02-07 04:23:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:23:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:23:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 23, 58, 732978), True, datetime.datetime(2025, 2, 7, 4, 23, 58, 732978), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:23:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:23:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:23:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:23:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:23:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:14.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 04:23:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:14 UTC)" (scheduled at 2025-02-07 04:23:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:23:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:23:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:23:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:23:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:23:59,235 [235] [DEBUG] [app] Starting request: urn:request:4ce03437-f4f3-42db-a2b1-dad93265c457 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:59,235 [238] [DEBUG] [app] Starting request: urn:request:6e5fb775-01a0-4e66-abe6-eafaa1fb6733 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:23:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:23:59,238 [245] [DEBUG] [app] Starting request: urn:request:8fe811e6-9dbe-4305-8886-3ecaf153353f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:59,238 [239] [DEBUG] [app] Starting request: urn:request:dd38ce89-b704-4501-a4e5-0fa30944e7cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:23:59,238 [239] [DEBUG] [app] Ending request: urn:request:dd38ce89-b704-4501-a4e5-0fa30944e7cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dd38ce89-b704-4501-a4e5-0fa30944e7cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:23:59,238 [245] [DEBUG] [app] Ending request: urn:request:8fe811e6-9dbe-4305-8886-3ecaf153353f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fe811e6-9dbe-4305-8886-3ecaf153353f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:23:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:23:59,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:23:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:23:59,241 [235] [DEBUG] [app] Starting request: urn:request:8e9d6618-54ce-4f79-8ecc-02be9722da90 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:59,241 [237] [DEBUG] [app] Starting request: urn:request:ea70b943-4778-437f-afdd-62ecb62b3d70 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:23:59,241 [235] [DEBUG] [app] Ending request: urn:request:8e9d6618-54ce-4f79-8ecc-02be9722da90 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e9d6618-54ce-4f79-8ecc-02be9722da90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:23:59,241 [237] [DEBUG] [app] Ending request: urn:request:ea70b943-4778-437f-afdd-62ecb62b3d70 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ea70b943-4778-437f-afdd-62ecb62b3d70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:23:59,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:23:59,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:59,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:23:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:23:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:23:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:23:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:23:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:23:59,245 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:23:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:23:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:23:59,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:23:59,252 [238] [DEBUG] [app] Ending request: urn:request:6e5fb775-01a0-4e66-abe6-eafaa1fb6733 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6e5fb775-01a0-4e66-abe6-eafaa1fb6733', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:59,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:23:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:23:59,253 [235] [DEBUG] [app] Ending request: urn:request:4ce03437-f4f3-42db-a2b1-dad93265c457 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ce03437-f4f3-42db-a2b1-dad93265c457', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:23:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:23:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:23:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:23:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 04:23:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:23:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:04.936688+00:00 (in 4.998308 seconds)
exportactionlogsworker stdout | 2025-02-07 04:23:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:59 UTC)" (scheduled at 2025-02-07 04:23:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:23:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:23:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:24:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:24:00,037 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:24:01,176 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:24:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:24:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:23.420886+00:00 (in 20.001315 seconds)
repositorygcworker stdout | 2025-02-07 04:24:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:03 UTC)" (scheduled at 2025-02-07 04:24:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:24:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:24:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 3, 419842), True, datetime.datetime(2025, 2, 7, 4, 24, 3, 419842), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:24:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:24:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:24:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:24:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:24:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:33.919286+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2025-02-07 04:24:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:33 UTC)" (scheduled at 2025-02-07 04:24:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:24:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 24, 3, 920074), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:24:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:24:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:24:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:24:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:24:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:52.937648+00:00 (in 48.000534 seconds)
exportactionlogsworker stdout | 2025-02-07 04:24:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:04 UTC)" (scheduled at 2025-02-07 04:24:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:24:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:24:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 4, 937407), True, datetime.datetime(2025, 2, 7, 4, 24, 4, 937407), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:24:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:24:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:24:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:24:05,484 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:24:06,579 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:24:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:24:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:18.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:24:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:18 UTC)" (scheduled at 2025-02-07 04:24:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:24:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:24:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 8, 732887), True, datetime.datetime(2025, 2, 7, 4, 24, 8, 732887), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:24:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:24:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:24:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:18 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:24:10,418 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
gcworker stdout | 2025-02-07 04:24:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:24:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:24.423055+00:00 (in 13.998414 seconds)
gcworker stdout | 2025-02-07 04:24:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:40 UTC)" (scheduled at 2025-02-07 04:24:10.424250+00:00)
gcworker stdout | 2025-02-07 04:24:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:24:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901950430, None, 1, 0])
gcworker stdout | 2025-02-07 04:24:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:24:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:40 UTC)" executed successfully
builder stdout | 2025-02-07 04:24:11,124 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:24:11,558 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:24:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:24:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:41.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 04:24:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:41 UTC)" (scheduled at 2025-02-07 04:24:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:24:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898651836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:24:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:24:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:24:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:24:12,863 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:24:13,781 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:24:13,868 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:24:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:24:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:29.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 04:24:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:29 UTC)" (scheduled at 2025-02-07 04:24:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:24:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:24:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:24:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:24:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:24:14,235 [235] [DEBUG] [app] Starting request: urn:request:75657e52-aeaf-4a8c-a061-db1b24a854b0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:14,235 [238] [DEBUG] [app] Starting request: urn:request:a4a5b1ea-ddb8-49b1-aaeb-5703be66aa18 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:24:14,238 [244] [DEBUG] [app] Starting request: urn:request:304d9024-3242-4337-9e48-02804638d05b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:14,238 [243] [DEBUG] [app] Starting request: urn:request:505a3f5e-bc85-4cd8-839c-e88fc2d46c80 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:14,239 [243] [DEBUG] [app] Ending request: urn:request:505a3f5e-bc85-4cd8-839c-e88fc2d46c80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:505a3f5e-bc85-4cd8-839c-e88fc2d46c80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:24:14,239 [244] [DEBUG] [app] Ending request: urn:request:304d9024-3242-4337-9e48-02804638d05b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:304d9024-3242-4337-9e48-02804638d05b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:24:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:24:14,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:14,242 [235] [DEBUG] [app] Starting request: urn:request:2c2bdf74-7ccf-4796-af51-95911d0c9fd8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:24:14,242 [235] [DEBUG] [app] Ending request: urn:request:2c2bdf74-7ccf-4796-af51-95911d0c9fd8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2c2bdf74-7ccf-4796-af51-95911d0c9fd8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:24:14,242 [237] [DEBUG] [app] Starting request: urn:request:a7bdaaef-4710-4958-862f-14126255826d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:24:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:14,242 [237] [DEBUG] [app] Ending request: urn:request:a7bdaaef-4710-4958-862f-14126255826d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7bdaaef-4710-4958-862f-14126255826d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:24:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:24:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:14,253 [235] [DEBUG] [app] Ending request: urn:request:75657e52-aeaf-4a8c-a061-db1b24a854b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75657e52-aeaf-4a8c-a061-db1b24a854b0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 04:24:14,254 [238] [DEBUG] [app] Ending request: urn:request:a4a5b1ea-ddb8-49b1-aaeb-5703be66aa18 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a4a5b1ea-ddb8-49b1-aaeb-5703be66aa18', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:24:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:24:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
autopruneworker stdout | 2025-02-07 04:24:14,896 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:15,399 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:24:15,906 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:24:15,980 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:24:16,079 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:24:16,458 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:24:16,670 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:24:16,672 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:24:16,676 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:24:16,678 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:24:16,682 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:24:16,917 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:24:17,020 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:24:17,063 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:24:17,696 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:24:17,700 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:24:17,702 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:24:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:24:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:28.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 04:24:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:28 UTC)" (scheduled at 2025-02-07 04:24:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:24:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:24:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 18, 732914), True, datetime.datetime(2025, 2, 7, 4, 24, 18, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:24:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:24:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:24:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:28 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:24:18,769 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:24:18,844 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:24:18,863 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:24:18,867 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:24:18,871 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:24:18,873 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:24:18,876 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:24:18,880 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:24:18,884 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:24:18,886 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:24:18,889 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:24:19,244 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:24:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:24:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:03.419110+00:00 (in 39.997760 seconds)
repositorygcworker stdout | 2025-02-07 04:24:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:23 UTC)" (scheduled at 2025-02-07 04:24:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:24:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:24:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:24:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:24:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:40.424250+00:00 (in 16.000759 seconds)
gcworker stdout | 2025-02-07 04:24:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:54 UTC)" (scheduled at 2025-02-07 04:24:24.423055+00:00)
gcworker stdout | 2025-02-07 04:24:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:24:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:24:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:24:28,242 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:24:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:24:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:38.732147+00:00 (in 9.999473 seconds)
notificationworker stdout | 2025-02-07 04:24:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:38 UTC)" (scheduled at 2025-02-07 04:24:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:24:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:24:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 28, 732999), True, datetime.datetime(2025, 2, 7, 4, 24, 28, 732999), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:24:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:24:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:24:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:24:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:24:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:44.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 04:24:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:44 UTC)" (scheduled at 2025-02-07 04:24:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:24:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:24:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:24:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:24:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:24:29,235 [235] [DEBUG] [app] Starting request: urn:request:a38d5fcd-8994-4c28-a5d9-09f9c81b95a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:29,235 [238] [DEBUG] [app] Starting request: urn:request:845a5a14-7f00-4e72-9854-ea17be06243b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:24:29,238 [246] [DEBUG] [app] Starting request: urn:request:652ff787-fcf7-470a-ba65-58f0f3a191ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:29,238 [239] [DEBUG] [app] Starting request: urn:request:58fd32b2-c3c9-47b1-92ae-d3d9d4a8cb04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:29,239 [246] [DEBUG] [app] Ending request: urn:request:652ff787-fcf7-470a-ba65-58f0f3a191ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:652ff787-fcf7-470a-ba65-58f0f3a191ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:24:29,239 [239] [DEBUG] [app] Ending request: urn:request:58fd32b2-c3c9-47b1-92ae-d3d9d4a8cb04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58fd32b2-c3c9-47b1-92ae-d3d9d4a8cb04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:24:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:24:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:29,242 [238] [DEBUG] [app] Starting request: urn:request:1b528038-46cc-415b-b298-a29c26cce547 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:24:29,242 [237] [DEBUG] [app] Starting request: urn:request:5605a64f-5dec-41ad-90ca-49ff6137dd81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:24:29,242 [238] [DEBUG] [app] Ending request: urn:request:1b528038-46cc-415b-b298-a29c26cce547 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1b528038-46cc-415b-b298-a29c26cce547', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:24:29,242 [237] [DEBUG] [app] Ending request: urn:request:5605a64f-5dec-41ad-90ca-49ff6137dd81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5605a64f-5dec-41ad-90ca-49ff6137dd81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:24:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:24:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:29,254 [238] [DEBUG] [app] Ending request: urn:request:845a5a14-7f00-4e72-9854-ea17be06243b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:845a5a14-7f00-4e72-9854-ea17be06243b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:24:29,255 [235] [DEBUG] [app] Ending request: urn:request:a38d5fcd-8994-4c28-a5d9-09f9c81b95a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a38d5fcd-8994-4c28-a5d9-09f9c81b95a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:24:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:24:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:24:30,066 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:24:31,211 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:24:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:24:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:32.625945+00:00 (in 1.004367 seconds)
namespacegcworker stdout | 2025-02-07 04:24:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:31 UTC)" (scheduled at 2025-02-07 04:24:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:24:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:24:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 31, 621867), True, datetime.datetime(2025, 2, 7, 4, 24, 31, 621867), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:24:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:24:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:24:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:24:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:24:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:31.621131+00:00 (in 58.994704 seconds)
namespacegcworker stdout | 2025-02-07 04:24:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:32 UTC)" (scheduled at 2025-02-07 04:24:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:24:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:24:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:24:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:24:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:03.919286+00:00 (in 29.999495 seconds)
buildlogsarchiver stdout | 2025-02-07 04:24:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:03 UTC)" (scheduled at 2025-02-07 04:24:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:24:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 24, 33, 920054), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:24:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:24:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:24:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:24:35,520 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:24:36,589 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:56.713547+00:00 (in 18.997139 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:37 UTC)" (scheduled at 2025-02-07 04:24:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:24:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:24:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:48.732147+00:00 (in 9.999513 seconds)
notificationworker stdout | 2025-02-07 04:24:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:48 UTC)" (scheduled at 2025-02-07 04:24:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:24:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:24:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 38, 732919), True, datetime.datetime(2025, 2, 7, 4, 24, 38, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:24:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:24:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:24:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:24:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:24:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:54.423055+00:00 (in 13.998311 seconds)
gcworker stdout | 2025-02-07 04:24:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:10 UTC)" (scheduled at 2025-02-07 04:24:40.424250+00:00)
gcworker stdout | 2025-02-07 04:24:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:24:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738901980431, None, 1, 0])
gcworker stdout | 2025-02-07 04:24:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:24:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:24:40,446 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:24:41,141 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:24:41,578 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:24:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:24:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:11.831521+00:00 (in 29.999530 seconds)
autopruneworker stdout | 2025-02-07 04:24:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:11 UTC)" (scheduled at 2025-02-07 04:24:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:24:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898681836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:24:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:24:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:24:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:24:42,887 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:24:43,817 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:24:43,901 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:24:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:24:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:59.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 04:24:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:59 UTC)" (scheduled at 2025-02-07 04:24:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:24:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:24:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:24:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:24:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:24:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:24:44,235 [235] [DEBUG] [app] Starting request: urn:request:b6e477a8-c790-4713-af75-be8e116abf16 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:44,235 [238] [DEBUG] [app] Starting request: urn:request:d1f0e81c-ee3c-41a8-87a3-f5fa2c125c3e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:24:44,238 [244] [DEBUG] [app] Starting request: urn:request:51a2aeec-d466-4613-bb89-95f65c24a0c4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:44,238 [239] [DEBUG] [app] Starting request: urn:request:23f34d2c-a9eb-41c7-8bd0-d268b7e72a6b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:44,238 [239] [DEBUG] [app] Ending request: urn:request:23f34d2c-a9eb-41c7-8bd0-d268b7e72a6b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:23f34d2c-a9eb-41c7-8bd0-d268b7e72a6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:24:44,238 [244] [DEBUG] [app] Ending request: urn:request:51a2aeec-d466-4613-bb89-95f65c24a0c4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:51a2aeec-d466-4613-bb89-95f65c24a0c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:24:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:24:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:44,241 [237] [DEBUG] [app] Starting request: urn:request:d99fc65f-abcc-4b23-9b6b-724e26831207 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:24:44,241 [237] [DEBUG] [app] Ending request: urn:request:d99fc65f-abcc-4b23-9b6b-724e26831207 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d99fc65f-abcc-4b23-9b6b-724e26831207', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:24:44,242 [235] [DEBUG] [app] Starting request: urn:request:c9622197-0af7-48e7-97e2-8950fff64ae6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:24:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:44,242 [235] [DEBUG] [app] Ending request: urn:request:c9622197-0af7-48e7-97e2-8950fff64ae6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9622197-0af7-48e7-97e2-8950fff64ae6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:24:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:24:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:44,254 [235] [DEBUG] [app] Ending request: urn:request:b6e477a8-c790-4713-af75-be8e116abf16 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6e477a8-c790-4713-af75-be8e116abf16', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:44,254 [238] [DEBUG] [app] Ending request: urn:request:d1f0e81c-ee3c-41a8-87a3-f5fa2c125c3e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1f0e81c-ee3c-41a8-87a3-f5fa2c125c3e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:24:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 04:24:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:24:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:24:44,922 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:45,434 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:24:45,934 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:24:46,011 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:24:46,089 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:24:46,494 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:24:46,678 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:24:46,682 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:24:46,685 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:24:46,688 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:24:46,692 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:24:46,946 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:24:47,056 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:24:47,089 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:24:47,704 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:24:47,708 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:24:47,710 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:24:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:24:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:50.814677+00:00 (in 2.082098 seconds)
notificationworker stdout | 2025-02-07 04:24:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:58 UTC)" (scheduled at 2025-02-07 04:24:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:24:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:24:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 48, 732848), True, datetime.datetime(2025, 2, 7, 4, 24, 48, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:24:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:24:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:24:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:24:58 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:24:48,804 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-registry stdout | 2025-02-07 04:24:48,873 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:24:48,876 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:24:48,879 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
buildlogsarchiver stdout | 2025-02-07 04:24:48,881 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
gunicorn-registry stdout | 2025-02-07 04:24:48,885 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:24:48,888 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:24:48,891 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:24:48,893 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:24:48,896 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:24:48,900 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
blobuploadcleanupworker stdout | 2025-02-07 04:24:49,258 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:24:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:24:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:58.732147+00:00 (in 7.917005 seconds)
notificationworker stdout | 2025-02-07 04:24:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:50 UTC)" (scheduled at 2025-02-07 04:24:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:24:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:24:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:24:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:24:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:24:59.937899+00:00 (in 6.999809 seconds)
exportactionlogsworker stdout | 2025-02-07 04:24:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:29:52 UTC)" (scheduled at 2025-02-07 04:24:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:24:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 24, 52, 938351), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:24:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 24, 52, 938351), True, datetime.datetime(2025, 2, 7, 4, 24, 52, 938351), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:24:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 24, 52, 938351), True, datetime.datetime(2025, 2, 7, 4, 24, 52, 938351), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 24, 52, 938351), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:24:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:24:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:29:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:24:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:24:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:10.424250+00:00 (in 16.000712 seconds)
gcworker stdout | 2025-02-07 04:24:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:24 UTC)" (scheduled at 2025-02-07 04:24:54.423055+00:00)
gcworker stdout | 2025-02-07 04:24:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:24:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:24:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:24:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:56.118855+00:00 (in 59.999486 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:24:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:56 UTC)" (scheduled at 2025-02-07 04:24:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:24:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:24:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:24:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:37.715878+00:00 (in 41.001885 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:56 UTC)" (scheduled at 2025-02-07 04:24:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:24:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:24:58,274 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:24:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:24:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:08.732147+00:00 (in 9.999539 seconds)
notificationworker stdout | 2025-02-07 04:24:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:08 UTC)" (scheduled at 2025-02-07 04:24:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:24:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:24:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 24, 58, 732890), True, datetime.datetime(2025, 2, 7, 4, 24, 58, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:24:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:24:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:24:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:24:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:24:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:14.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 04:24:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:14 UTC)" (scheduled at 2025-02-07 04:24:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:24:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:24:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:24:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:24:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:24:59,236 [235] [DEBUG] [app] Starting request: urn:request:fadf0840-3e7f-46da-a59e-394314560808 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:59,236 [237] [DEBUG] [app] Starting request: urn:request:f30345c6-7eff-461d-9cbd-c8b08e8866bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:24:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:24:59,239 [246] [DEBUG] [app] Starting request: urn:request:0f66989d-8139-45d4-8f90-dd6f0797a0e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:59,239 [243] [DEBUG] [app] Starting request: urn:request:476256a7-5e9c-43fa-87f3-e8f3cfcb864e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:24:59,239 [246] [DEBUG] [app] Ending request: urn:request:0f66989d-8139-45d4-8f90-dd6f0797a0e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f66989d-8139-45d4-8f90-dd6f0797a0e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:24:59,239 [243] [DEBUG] [app] Ending request: urn:request:476256a7-5e9c-43fa-87f3-e8f3cfcb864e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:476256a7-5e9c-43fa-87f3-e8f3cfcb864e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:24:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:24:59,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:24:59,242 [235] [DEBUG] [app] Starting request: urn:request:c3b1b21c-309a-4d43-842c-270bff33ea21 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:24:59,242 [238] [DEBUG] [app] Starting request: urn:request:ef9cc63c-03e8-4e14-a2b4-5fb4b26f3b7c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:24:59,242 [235] [DEBUG] [app] Ending request: urn:request:c3b1b21c-309a-4d43-842c-270bff33ea21 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c3b1b21c-309a-4d43-842c-270bff33ea21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:24:59,243 [238] [DEBUG] [app] Ending request: urn:request:ef9cc63c-03e8-4e14-a2b4-5fb4b26f3b7c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef9cc63c-03e8-4e14-a2b4-5fb4b26f3b7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:24:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:24:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:24:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:24:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:24:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:24:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:24:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:24:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:24:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:24:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:24:59,254 [237] [DEBUG] [app] Ending request: urn:request:f30345c6-7eff-461d-9cbd-c8b08e8866bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f30345c6-7eff-461d-9cbd-c8b08e8866bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:24:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:24:59,255 [235] [DEBUG] [app] Ending request: urn:request:fadf0840-3e7f-46da-a59e-394314560808 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fadf0840-3e7f-46da-a59e-394314560808', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:24:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:24:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:24:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:24:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:24:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:24:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:04.936688+00:00 (in 4.998287 seconds)
exportactionlogsworker stdout | 2025-02-07 04:24:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:59 UTC)" (scheduled at 2025-02-07 04:24:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:24:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:24:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:25:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:25:00,089 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:25:01,230 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:25:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:25:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:23.420886+00:00 (in 20.001311 seconds)
repositorygcworker stdout | 2025-02-07 04:25:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:03 UTC)" (scheduled at 2025-02-07 04:25:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:25:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:25:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 3, 419842), True, datetime.datetime(2025, 2, 7, 4, 25, 3, 419842), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:25:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:25:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:25:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:25:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:25:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:33.919286+00:00 (in 29.999560 seconds)
buildlogsarchiver stdout | 2025-02-07 04:25:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:33 UTC)" (scheduled at 2025-02-07 04:25:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:25:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 25, 3, 919971), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:25:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:25:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:25:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:25:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:25:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:59.937899+00:00 (in 55.000706 seconds)
exportactionlogsworker stdout | 2025-02-07 04:25:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:04 UTC)" (scheduled at 2025-02-07 04:25:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:25:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:25:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 4, 937525), True, datetime.datetime(2025, 2, 7, 4, 25, 4, 937525), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:25:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:25:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:25:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:25:05,556 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:25:06,625 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:25:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:25:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:18.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 04:25:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:18 UTC)" (scheduled at 2025-02-07 04:25:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:25:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:25:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 8, 732861), True, datetime.datetime(2025, 2, 7, 4, 25, 8, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:25:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:25:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:25:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:25:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:25:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:24.423055+00:00 (in 13.998355 seconds)
gcworker stdout | 2025-02-07 04:25:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:40 UTC)" (scheduled at 2025-02-07 04:25:10.424250+00:00)
gcworker stdout | 2025-02-07 04:25:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:25:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902010431, None, 1, 0])
gcworker stdout | 2025-02-07 04:25:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:25:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:25:10,479 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:25:11,162 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:25:11,588 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:25:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:25:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:41.831521+00:00 (in 29.999559 seconds)
autopruneworker stdout | 2025-02-07 04:25:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:41 UTC)" (scheduled at 2025-02-07 04:25:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:25:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898711836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:25:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:25:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:25:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:25:12,923 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:25:13,854 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:25:13,937 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:25:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:25:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:29.225378+00:00 (in 14.999532 seconds)
quotatotalworker stdout | 2025-02-07 04:25:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:29 UTC)" (scheduled at 2025-02-07 04:25:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:25:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:25:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:25:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:25:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:25:14,235 [237] [DEBUG] [app] Starting request: urn:request:e21e83fb-8387-4437-8012-55d710fdfc58 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:14,235 [235] [DEBUG] [app] Starting request: urn:request:33793fa6-ba9a-4ec4-8b41-612a7b09562d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:25:14,238 [239] [DEBUG] [app] Starting request: urn:request:86074c0c-ede5-4bb3-af63-349168c31baa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:14,239 [246] [DEBUG] [app] Starting request: urn:request:1aa66c8e-cb26-44ca-8d8a-f4db5226c2f5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:14,239 [246] [DEBUG] [app] Ending request: urn:request:1aa66c8e-cb26-44ca-8d8a-f4db5226c2f5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1aa66c8e-cb26-44ca-8d8a-f4db5226c2f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:14,239 [239] [DEBUG] [app] Ending request: urn:request:86074c0c-ede5-4bb3-af63-349168c31baa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86074c0c-ede5-4bb3-af63-349168c31baa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:25:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:25:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:14,242 [237] [DEBUG] [app] Starting request: urn:request:9b7e3104-b417-4010-acf8-8ea17e5a1347 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:14,242 [238] [DEBUG] [app] Starting request: urn:request:db1cb4ca-265c-4d31-9c9e-1b8835c0d51e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:14,243 [237] [DEBUG] [app] Ending request: urn:request:9b7e3104-b417-4010-acf8-8ea17e5a1347 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9b7e3104-b417-4010-acf8-8ea17e5a1347', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:25:14,243 [238] [DEBUG] [app] Ending request: urn:request:db1cb4ca-265c-4d31-9c9e-1b8835c0d51e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db1cb4ca-265c-4d31-9c9e-1b8835c0d51e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:25:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:25:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:25:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:14,255 [237] [DEBUG] [app] Ending request: urn:request:e21e83fb-8387-4437-8012-55d710fdfc58 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e21e83fb-8387-4437-8012-55d710fdfc58', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:14,255 [235] [DEBUG] [app] Ending request: urn:request:33793fa6-ba9a-4ec4-8b41-612a7b09562d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:33793fa6-ba9a-4ec4-8b41-612a7b09562d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:25:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:25:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:25:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:25:14,958 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:15,465 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:25:15,966 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:25:16,022 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:25:16,125 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:25:16,531 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:25:16,687 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:25:16,689 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:25:16,692 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:25:16,695 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:25:16,699 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:25:16,969 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:25:17,093 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:25:17,126 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:25:17,712 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:25:17,716 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:25:17,719 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:25:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:25:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:28.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 04:25:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:28 UTC)" (scheduled at 2025-02-07 04:25:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:25:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:25:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 18, 732859), True, datetime.datetime(2025, 2, 7, 4, 25, 18, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:25:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:25:18,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:25:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:28 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:25:18,840 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-registry stdout | 2025-02-07 04:25:18,882 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:25:18,885 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:25:18,890 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:25:18,894 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:25:18,898 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:25:18,901 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:25:18,904 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:25:18,907 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:25:18,909 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:25:18,916 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:25:19,291 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:25:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:25:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:03.419110+00:00 (in 39.997746 seconds)
repositorygcworker stdout | 2025-02-07 04:25:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:23 UTC)" (scheduled at 2025-02-07 04:25:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:25:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:25:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:25:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:25:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:40.424250+00:00 (in 16.000747 seconds)
gcworker stdout | 2025-02-07 04:25:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:54 UTC)" (scheduled at 2025-02-07 04:25:24.423055+00:00)
gcworker stdout | 2025-02-07 04:25:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:25:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:25:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:25:28,311 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:25:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:25:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:38.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 04:25:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:38 UTC)" (scheduled at 2025-02-07 04:25:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:25:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:25:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 28, 732880), True, datetime.datetime(2025, 2, 7, 4, 25, 28, 732880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:25:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:25:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:25:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:25:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:25:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:44.225378+00:00 (in 14.999558 seconds)
quotatotalworker stdout | 2025-02-07 04:25:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:44 UTC)" (scheduled at 2025-02-07 04:25:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:25:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:25:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:25:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:25:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:25:29,235 [237] [DEBUG] [app] Starting request: urn:request:2903e3a6-6643-4a80-96e1-43134f7462bd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:29,235 [236] [DEBUG] [app] Starting request: urn:request:d3ca8a5b-1eb1-4160-8b8b-efc16dd683cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:25:29,238 [245] [DEBUG] [app] Starting request: urn:request:c368ec1c-5d2c-40e1-a34c-fafd87d07528 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:29,238 [246] [DEBUG] [app] Starting request: urn:request:6e3e4b49-a7ef-4671-9051-7284fc4ddc4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:29,238 [245] [DEBUG] [app] Ending request: urn:request:c368ec1c-5d2c-40e1-a34c-fafd87d07528 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c368ec1c-5d2c-40e1-a34c-fafd87d07528', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:29,238 [246] [DEBUG] [app] Ending request: urn:request:6e3e4b49-a7ef-4671-9051-7284fc4ddc4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e3e4b49-a7ef-4671-9051-7284fc4ddc4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:25:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:25:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:29,241 [236] [DEBUG] [app] Starting request: urn:request:07ba8a57-47a2-4b0f-b281-5e78196b574f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:29,241 [235] [DEBUG] [app] Starting request: urn:request:945fd86b-cc8f-4f95-bf97-36f35eed7bb8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:29,241 [236] [DEBUG] [app] Ending request: urn:request:07ba8a57-47a2-4b0f-b281-5e78196b574f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:07ba8a57-47a2-4b0f-b281-5e78196b574f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:25:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:25:29,242 [235] [DEBUG] [app] Ending request: urn:request:945fd86b-cc8f-4f95-bf97-36f35eed7bb8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:945fd86b-cc8f-4f95-bf97-36f35eed7bb8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:25:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:25:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:29,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:29,254 [237] [DEBUG] [app] Ending request: urn:request:2903e3a6-6643-4a80-96e1-43134f7462bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2903e3a6-6643-4a80-96e1-43134f7462bd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:29,254 [236] [DEBUG] [app] Ending request: urn:request:d3ca8a5b-1eb1-4160-8b8b-efc16dd683cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d3ca8a5b-1eb1-4160-8b8b-efc16dd683cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:25:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:25:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:25:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:25:30,113 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:25:31,261 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:25:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:25:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:32.625945+00:00 (in 1.004358 seconds)
namespacegcworker stdout | 2025-02-07 04:25:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:31 UTC)" (scheduled at 2025-02-07 04:25:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:25:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:25:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 31, 621895), True, datetime.datetime(2025, 2, 7, 4, 25, 31, 621895), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:25:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:25:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:25:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:25:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:25:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:31.621131+00:00 (in 58.994724 seconds)
namespacegcworker stdout | 2025-02-07 04:25:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:32 UTC)" (scheduled at 2025-02-07 04:25:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:25:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:25:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:25:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:25:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:03.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 04:25:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:03 UTC)" (scheduled at 2025-02-07 04:25:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:25:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 25, 33, 919983), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:25:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:25:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:25:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:25:35,579 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:25:36,639 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:56.713547+00:00 (in 18.997203 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:37 UTC)" (scheduled at 2025-02-07 04:25:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:25:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:25:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:48.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 04:25:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:48 UTC)" (scheduled at 2025-02-07 04:25:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:25:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:25:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 38, 732857), True, datetime.datetime(2025, 2, 7, 4, 25, 38, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:25:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:25:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:25:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:25:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:25:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:54.423055+00:00 (in 13.998371 seconds)
gcworker stdout | 2025-02-07 04:25:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:10 UTC)" (scheduled at 2025-02-07 04:25:40.424250+00:00)
gcworker stdout | 2025-02-07 04:25:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:25:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902040430, None, 1, 0])
gcworker stdout | 2025-02-07 04:25:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:25:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:25:40,511 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:25:41,174 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:25:41,625 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:25:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:25:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:11.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 04:25:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:11 UTC)" (scheduled at 2025-02-07 04:25:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:25:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898741836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:25:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:25:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:25:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:25:42,959 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:25:43,887 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:25:43,963 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:25:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:25:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:59.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 04:25:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:59 UTC)" (scheduled at 2025-02-07 04:25:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:25:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:25:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:25:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:25:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:25:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:25:44,235 [235] [DEBUG] [app] Starting request: urn:request:a4d1e270-776b-40bd-ae65-78682d9abeed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:44,235 [238] [DEBUG] [app] Starting request: urn:request:47144dbf-5d93-46aa-89ca-cafa6ee50f6e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:25:44,239 [239] [DEBUG] [app] Starting request: urn:request:958d1267-3591-48fc-a229-69a5b74232c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:44,239 [246] [DEBUG] [app] Starting request: urn:request:3ef1d6ab-ca4f-4371-9145-40d226e31989 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:44,239 [239] [DEBUG] [app] Ending request: urn:request:958d1267-3591-48fc-a229-69a5b74232c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:958d1267-3591-48fc-a229-69a5b74232c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:44,239 [246] [DEBUG] [app] Ending request: urn:request:3ef1d6ab-ca4f-4371-9145-40d226e31989 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ef1d6ab-ca4f-4371-9145-40d226e31989', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:25:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:25:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:44,243 [236] [DEBUG] [app] Starting request: urn:request:4b496925-18b6-4210-a1d6-5190e7d9c08f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:44,243 [237] [DEBUG] [app] Starting request: urn:request:b8010979-1723-46ec-a964-f22828d43639 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:44,243 [236] [DEBUG] [app] Ending request: urn:request:4b496925-18b6-4210-a1d6-5190e7d9c08f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b496925-18b6-4210-a1d6-5190e7d9c08f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:25:44,243 [237] [DEBUG] [app] Ending request: urn:request:b8010979-1723-46ec-a964-f22828d43639 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8010979-1723-46ec-a964-f22828d43639', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:25:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:25:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:25:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:44,255 [235] [DEBUG] [app] Ending request: urn:request:a4d1e270-776b-40bd-ae65-78682d9abeed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a4d1e270-776b-40bd-ae65-78682d9abeed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:25:44,255 [238] [DEBUG] [app] Ending request: urn:request:47144dbf-5d93-46aa-89ca-cafa6ee50f6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47144dbf-5d93-46aa-89ca-cafa6ee50f6e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:25:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:25:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
autopruneworker stdout | 2025-02-07 04:25:44,995 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:45,502 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:25:45,978 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:25:46,042 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:25:46,162 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:25:46,559 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:25:46,695 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:25:46,697 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:25:46,700 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:25:46,704 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:25:46,706 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:25:47,005 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:25:47,129 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:25:47,162 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:25:47,719 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:25:47,723 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:25:47,726 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:25:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:25:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:50.814677+00:00 (in 2.082091 seconds)
notificationworker stdout | 2025-02-07 04:25:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:58 UTC)" (scheduled at 2025-02-07 04:25:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:25:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:25:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 48, 732908), True, datetime.datetime(2025, 2, 7, 4, 25, 48, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:25:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:25:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:25:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:25:58 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:25:48,853 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-registry stdout | 2025-02-07 04:25:48,890 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:25:48,895 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:25:48,901 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:25:48,905 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:25:48,908 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:25:48,912 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:25:48,914 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:25:48,917 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:25:48,920 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:25:48,951 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:25:49,327 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:25:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:25:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:25:58.732147+00:00 (in 7.917002 seconds)
notificationworker stdout | 2025-02-07 04:25:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:50 UTC)" (scheduled at 2025-02-07 04:25:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:25:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:25:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:25:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:25:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:10.424250+00:00 (in 16.000740 seconds)
gcworker stdout | 2025-02-07 04:25:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:24 UTC)" (scheduled at 2025-02-07 04:25:54.423055+00:00)
gcworker stdout | 2025-02-07 04:25:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:25:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:25:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:25:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:56.118855+00:00 (in 59.999534 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:25:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:56 UTC)" (scheduled at 2025-02-07 04:25:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:25:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:25:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:25:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:37.715878+00:00 (in 41.001879 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:56 UTC)" (scheduled at 2025-02-07 04:25:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:25:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:25:58,343 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:25:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:25:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:08.732147+00:00 (in 9.999513 seconds)
notificationworker stdout | 2025-02-07 04:25:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:08 UTC)" (scheduled at 2025-02-07 04:25:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:25:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:25:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 25, 58, 732906), True, datetime.datetime(2025, 2, 7, 4, 25, 58, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:25:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:25:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:25:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:25:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:25:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:14.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 04:25:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:14 UTC)" (scheduled at 2025-02-07 04:25:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:25:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:25:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:25:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:25:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:25:59,235 [236] [DEBUG] [app] Starting request: urn:request:7e4f3555-0177-48b1-ad37-06551747c048 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:59,235 [235] [DEBUG] [app] Starting request: urn:request:13f09354-eff2-4eb1-8d20-4bc19c969535 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:25:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:25:59,239 [243] [DEBUG] [app] Starting request: urn:request:253cde19-fa22-4f9d-9f2e-1f25fe6f96ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:59,239 [244] [DEBUG] [app] Starting request: urn:request:41e53dad-2e79-4bac-98a2-571239d12052 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:25:59,240 [243] [DEBUG] [app] Ending request: urn:request:253cde19-fa22-4f9d-9f2e-1f25fe6f96ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:253cde19-fa22-4f9d-9f2e-1f25fe6f96ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:59,240 [244] [DEBUG] [app] Ending request: urn:request:41e53dad-2e79-4bac-98a2-571239d12052 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:41e53dad-2e79-4bac-98a2-571239d12052', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:25:59,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:25:59,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:25:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:59,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:25:59,243 [235] [DEBUG] [app] Starting request: urn:request:b32f06a8-9bc4-4470-a88f-a3446cdbcec0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:59,243 [237] [DEBUG] [app] Starting request: urn:request:06b97edd-342d-470a-9cde-d32c7d98b859 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:25:59,243 [235] [DEBUG] [app] Ending request: urn:request:b32f06a8-9bc4-4470-a88f-a3446cdbcec0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b32f06a8-9bc4-4470-a88f-a3446cdbcec0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:25:59,243 [237] [DEBUG] [app] Ending request: urn:request:06b97edd-342d-470a-9cde-d32c7d98b859 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06b97edd-342d-470a-9cde-d32c7d98b859', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:25:59,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:25:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:25:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:25:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:25:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:25:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:25:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:25:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:25:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:25:59,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:25:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:25:59,256 [235] [DEBUG] [app] Ending request: urn:request:13f09354-eff2-4eb1-8d20-4bc19c969535 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13f09354-eff2-4eb1-8d20-4bc19c969535', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:59,256 [236] [DEBUG] [app] Ending request: urn:request:7e4f3555-0177-48b1-ad37-06551747c048 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e4f3555-0177-48b1-ad37-06551747c048', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:25:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:25:59,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:25:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:25:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:25:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:25:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
exportactionlogsworker stdout | 2025-02-07 04:25:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:25:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:04.936688+00:00 (in 4.998271 seconds)
exportactionlogsworker stdout | 2025-02-07 04:25:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:59 UTC)" (scheduled at 2025-02-07 04:25:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:25:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:25:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:26:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:26:00,129 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:26:01,298 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:26:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:26:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:23.420886+00:00 (in 20.001331 seconds)
repositorygcworker stdout | 2025-02-07 04:26:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:03 UTC)" (scheduled at 2025-02-07 04:26:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:26:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:26:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 3, 419828), True, datetime.datetime(2025, 2, 7, 4, 26, 3, 419828), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:26:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:26:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:26:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:26:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:26:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:33.919286+00:00 (in 29.999551 seconds)
buildlogsarchiver stdout | 2025-02-07 04:26:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:33 UTC)" (scheduled at 2025-02-07 04:26:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:26:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 26, 3, 919979), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:26:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:26:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:26:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:26:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:26:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:59.937899+00:00 (in 55.000771 seconds)
exportactionlogsworker stdout | 2025-02-07 04:26:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:04 UTC)" (scheduled at 2025-02-07 04:26:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:26:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:26:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 4, 937417), True, datetime.datetime(2025, 2, 7, 4, 26, 4, 937417), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:26:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:26:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:26:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:26:05,615 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:26:06,663 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:26:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:26:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:18.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 04:26:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:18 UTC)" (scheduled at 2025-02-07 04:26:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:26:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:26:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 8, 732873), True, datetime.datetime(2025, 2, 7, 4, 26, 8, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:26:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:26:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:26:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:26:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:26:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:24.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 04:26:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:40 UTC)" (scheduled at 2025-02-07 04:26:10.424250+00:00)
gcworker stdout | 2025-02-07 04:26:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:26:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902070430, None, 1, 0])
gcworker stdout | 2025-02-07 04:26:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:26:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:26:10,547 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:26:11,207 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:26:11,657 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:26:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:26:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:41.831521+00:00 (in 29.999519 seconds)
autopruneworker stdout | 2025-02-07 04:26:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:41 UTC)" (scheduled at 2025-02-07 04:26:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:26:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898771836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:26:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:26:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:26:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:26:12,995 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:26:13,908 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:26:13,975 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:26:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:26:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:29.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 04:26:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:29 UTC)" (scheduled at 2025-02-07 04:26:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:26:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:26:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:26:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:26:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:26:14,235 [235] [DEBUG] [app] Starting request: urn:request:a124e258-cee1-489d-9dec-ea875d377c54 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:14,235 [238] [DEBUG] [app] Starting request: urn:request:280c84a3-113d-4983-a86b-0fbeb2005b36 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:26:14,238 [246] [DEBUG] [app] Starting request: urn:request:58e65ad5-de0e-40d9-89a2-ef6070819621 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:14,238 [245] [DEBUG] [app] Starting request: urn:request:dae198d7-59a7-4fe2-a8dc-4230525d3ce0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:14,238 [246] [DEBUG] [app] Ending request: urn:request:58e65ad5-de0e-40d9-89a2-ef6070819621 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:58e65ad5-de0e-40d9-89a2-ef6070819621', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:14,238 [245] [DEBUG] [app] Ending request: urn:request:dae198d7-59a7-4fe2-a8dc-4230525d3ce0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dae198d7-59a7-4fe2-a8dc-4230525d3ce0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:14,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:26:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:26:14,238 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:26:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:14,241 [237] [DEBUG] [app] Starting request: urn:request:94720f6c-5b96-4bee-89ff-43d4d59b8627 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:14,241 [238] [DEBUG] [app] Starting request: urn:request:eb095f23-3fca-46bf-9a5a-d24108ad20c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:14,241 [237] [DEBUG] [app] Ending request: urn:request:94720f6c-5b96-4bee-89ff-43d4d59b8627 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:94720f6c-5b96-4bee-89ff-43d4d59b8627', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:26:14,241 [238] [DEBUG] [app] Ending request: urn:request:eb095f23-3fca-46bf-9a5a-d24108ad20c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb095f23-3fca-46bf-9a5a-d24108ad20c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:26:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:26:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:26:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:26:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:14,253 [238] [DEBUG] [app] Ending request: urn:request:280c84a3-113d-4983-a86b-0fbeb2005b36 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:280c84a3-113d-4983-a86b-0fbeb2005b36', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:26:14,253 [235] [DEBUG] [app] Ending request: urn:request:a124e258-cee1-489d-9dec-ea875d377c54 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a124e258-cee1-489d-9dec-ea875d377c54', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:26:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:26:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
autopruneworker stdout | 2025-02-07 04:26:15,026 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:15,512 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:26:16,014 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:26:16,049 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:26:16,198 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:26:16,595 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:26:16,705 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:26:16,708 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:26:16,711 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:26:16,713 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:26:16,716 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:26:17,034 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:26:17,166 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:26:17,199 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:26:17,726 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:26:17,731 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:26:17,734 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:26:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:26:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:28.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:26:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:28 UTC)" (scheduled at 2025-02-07 04:26:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:26:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:26:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 18, 732858), True, datetime.datetime(2025, 2, 7, 4, 26, 18, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:26:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:26:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:26:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:28 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:26:18,887 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-registry stdout | 2025-02-07 04:26:18,898 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:26:18,905 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:26:18,908 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:26:18,913 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:26:18,916 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:26:18,922 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:26:18,925 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:26:18,928 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:26:18,930 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:26:18,975 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:26:19,343 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:26:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:26:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:03.419110+00:00 (in 39.997733 seconds)
repositorygcworker stdout | 2025-02-07 04:26:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:23 UTC)" (scheduled at 2025-02-07 04:26:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:26:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:26:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:26:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:26:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:40.424250+00:00 (in 16.000763 seconds)
gcworker stdout | 2025-02-07 04:26:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:54 UTC)" (scheduled at 2025-02-07 04:26:24.423055+00:00)
gcworker stdout | 2025-02-07 04:26:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:26:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:26:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:26:28,353 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:26:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:26:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:38.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:26:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:38 UTC)" (scheduled at 2025-02-07 04:26:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:26:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:26:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 28, 732911), True, datetime.datetime(2025, 2, 7, 4, 26, 28, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:26:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:26:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:26:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:26:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:26:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:44.225378+00:00 (in 14.999521 seconds)
quotatotalworker stdout | 2025-02-07 04:26:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:44 UTC)" (scheduled at 2025-02-07 04:26:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:26:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:26:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:26:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:26:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:26:29,235 [237] [DEBUG] [app] Starting request: urn:request:5fc01a70-7dcc-4075-9ec4-42b6dfa1f907 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:29,235 [238] [DEBUG] [app] Starting request: urn:request:473eca2c-283f-4999-8a4e-a12fc6f99a75 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:26:29,238 [246] [DEBUG] [app] Starting request: urn:request:1397cba6-424f-4094-96cb-c249f81d96dc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:29,238 [244] [DEBUG] [app] Starting request: urn:request:f9fe2dec-7724-49b0-9c36-957364e56386 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:29,239 [246] [DEBUG] [app] Ending request: urn:request:1397cba6-424f-4094-96cb-c249f81d96dc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1397cba6-424f-4094-96cb-c249f81d96dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:29,239 [244] [DEBUG] [app] Ending request: urn:request:f9fe2dec-7724-49b0-9c36-957364e56386 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9fe2dec-7724-49b0-9c36-957364e56386', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:29,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:26:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:26:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:26:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:29,241 [238] [DEBUG] [app] Starting request: urn:request:d6d8aed6-1786-4f53-9430-5ebe15ea8f6a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:29,242 [235] [DEBUG] [app] Starting request: urn:request:41bc933d-8d1a-4713-be50-a62a5b53a621 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:29,242 [238] [DEBUG] [app] Ending request: urn:request:d6d8aed6-1786-4f53-9430-5ebe15ea8f6a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d6d8aed6-1786-4f53-9430-5ebe15ea8f6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:26:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:26:29,242 [235] [DEBUG] [app] Ending request: urn:request:41bc933d-8d1a-4713-be50-a62a5b53a621 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:41bc933d-8d1a-4713-be50-a62a5b53a621', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:26:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:26:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:29,253 [238] [DEBUG] [app] Ending request: urn:request:473eca2c-283f-4999-8a4e-a12fc6f99a75 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:473eca2c-283f-4999-8a4e-a12fc6f99a75', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:26:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:26:29,253 [237] [DEBUG] [app] Ending request: urn:request:5fc01a70-7dcc-4075-9ec4-42b6dfa1f907 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5fc01a70-7dcc-4075-9ec4-42b6dfa1f907', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:26:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:26:30,141 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:26:31,317 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:26:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:26:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:32.625945+00:00 (in 1.004377 seconds)
namespacegcworker stdout | 2025-02-07 04:26:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:31 UTC)" (scheduled at 2025-02-07 04:26:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:26:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:26:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 31, 621840), True, datetime.datetime(2025, 2, 7, 4, 26, 31, 621840), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:26:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:26:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:26:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:26:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:26:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:13.624301+00:00 (in 40.997875 seconds)
namespacegcworker stdout | 2025-02-07 04:26:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:32 UTC)" (scheduled at 2025-02-07 04:26:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:26:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:26:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:26:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:26:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:03.919286+00:00 (in 29.999503 seconds)
buildlogsarchiver stdout | 2025-02-07 04:26:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:03 UTC)" (scheduled at 2025-02-07 04:26:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:26:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 26, 33, 920075), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:26:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:26:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:26:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:26:35,638 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:26:36,700 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:56.713547+00:00 (in 18.997178 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:37 UTC)" (scheduled at 2025-02-07 04:26:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:26:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:26:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:48.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 04:26:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:48 UTC)" (scheduled at 2025-02-07 04:26:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:26:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:26:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 38, 732878), True, datetime.datetime(2025, 2, 7, 4, 26, 38, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:26:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:26:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:26:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:26:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:26:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:54.423055+00:00 (in 13.998387 seconds)
gcworker stdout | 2025-02-07 04:26:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:10 UTC)" (scheduled at 2025-02-07 04:26:40.424250+00:00)
gcworker stdout | 2025-02-07 04:26:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:26:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902100431, None, 1, 0])
gcworker stdout | 2025-02-07 04:26:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:26:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:26:40,583 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:26:41,230 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:26:41,691 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:26:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:26:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:11.831521+00:00 (in 29.999517 seconds)
autopruneworker stdout | 2025-02-07 04:26:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:11 UTC)" (scheduled at 2025-02-07 04:26:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:26:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898801836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:26:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:26:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:26:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:26:43,010 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:26:43,935 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:26:44,011 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:26:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:26:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:59.225378+00:00 (in 14.999498 seconds)
quotatotalworker stdout | 2025-02-07 04:26:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:59 UTC)" (scheduled at 2025-02-07 04:26:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:26:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:26:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:26:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:26:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:26:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:26:44,235 [237] [DEBUG] [app] Starting request: urn:request:1b92eee1-3b87-4b2e-92a4-f02d74e12053 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:44,235 [235] [DEBUG] [app] Starting request: urn:request:f5d5438f-d289-42a6-9d2f-84295dff3da3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:26:44,238 [239] [DEBUG] [app] Starting request: urn:request:d953f509-1c1b-4701-b0d5-5089a3d4d853 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:44,238 [245] [DEBUG] [app] Starting request: urn:request:20382961-3f3f-4424-b9a2-ea8b34398a8b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:44,238 [239] [DEBUG] [app] Ending request: urn:request:d953f509-1c1b-4701-b0d5-5089a3d4d853 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d953f509-1c1b-4701-b0d5-5089a3d4d853', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:44,238 [245] [DEBUG] [app] Ending request: urn:request:20382961-3f3f-4424-b9a2-ea8b34398a8b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20382961-3f3f-4424-b9a2-ea8b34398a8b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:26:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:26:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:44,241 [238] [DEBUG] [app] Starting request: urn:request:0218d32e-1fa0-4273-8684-b5f6652dc4f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:44,242 [238] [DEBUG] [app] Ending request: urn:request:0218d32e-1fa0-4273-8684-b5f6652dc4f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0218d32e-1fa0-4273-8684-b5f6652dc4f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:26:44,242 [237] [DEBUG] [app] Starting request: urn:request:1147c4fc-09ba-4e05-a906-16e69f64dc9d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:26:44,242 [237] [DEBUG] [app] Ending request: urn:request:1147c4fc-09ba-4e05-a906-16e69f64dc9d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1147c4fc-09ba-4e05-a906-16e69f64dc9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:26:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:26:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:44,253 [237] [DEBUG] [app] Ending request: urn:request:1b92eee1-3b87-4b2e-92a4-f02d74e12053 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1b92eee1-3b87-4b2e-92a4-f02d74e12053', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:26:44,253 [235] [DEBUG] [app] Ending request: urn:request:f5d5438f-d289-42a6-9d2f-84295dff3da3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f5d5438f-d289-42a6-9d2f-84295dff3da3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:26:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:26:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:26:45,062 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:45,549 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:26:46,050 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:26:46,060 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:26:46,234 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:26:46,631 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:26:46,713 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:26:46,716 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:26:46,719 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:26:46,721 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:26:46,725 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:26:47,070 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:26:47,204 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:26:47,214 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:26:47,734 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:26:47,737 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:26:47,740 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:26:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:26:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:50.814677+00:00 (in 2.082078 seconds)
notificationworker stdout | 2025-02-07 04:26:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:58 UTC)" (scheduled at 2025-02-07 04:26:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:26:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:26:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 48, 732918), True, datetime.datetime(2025, 2, 7, 4, 26, 48, 732918), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:26:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:26:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:26:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:26:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:26:48,908 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:26:48,914 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:26:48,916 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:26:48,921 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
namespacegcworker stdout | 2025-02-07 04:26:48,924 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gunicorn-registry stdout | 2025-02-07 04:26:48,927 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:26:48,931 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:26:48,934 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:26:48,938 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:26:48,941 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:26:49,010 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:26:49,353 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:26:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:26:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:26:58.732147+00:00 (in 7.916982 seconds)
notificationworker stdout | 2025-02-07 04:26:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:50 UTC)" (scheduled at 2025-02-07 04:26:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:26:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:26:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:26:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:26:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:10.424250+00:00 (in 16.000760 seconds)
gcworker stdout | 2025-02-07 04:26:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:24 UTC)" (scheduled at 2025-02-07 04:26:54.423055+00:00)
gcworker stdout | 2025-02-07 04:26:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:26:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:26:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:26:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:56.118855+00:00 (in 59.999532 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:26:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:56 UTC)" (scheduled at 2025-02-07 04:26:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:26:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:26:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:26:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:37.715878+00:00 (in 41.001841 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:56 UTC)" (scheduled at 2025-02-07 04:26:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:26:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:26:58,390 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:26:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:26:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:08.732147+00:00 (in 9.999427 seconds)
notificationworker stdout | 2025-02-07 04:26:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:08 UTC)" (scheduled at 2025-02-07 04:26:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:26:58,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:26:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 26, 58, 733034), True, datetime.datetime(2025, 2, 7, 4, 26, 58, 733034), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:26:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:26:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:26:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:26:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:26:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:14.225378+00:00 (in 14.999497 seconds)
quotatotalworker stdout | 2025-02-07 04:26:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:14 UTC)" (scheduled at 2025-02-07 04:26:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:26:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:26:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:26:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:26:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:26:59,235 [235] [DEBUG] [app] Starting request: urn:request:046f4e31-818b-4476-8e68-c1a7d88eacbd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:59,235 [236] [DEBUG] [app] Starting request: urn:request:6829b7e8-bc1f-48eb-9b52-b74149a9886d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:26:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:26:59,239 [246] [DEBUG] [app] Starting request: urn:request:31a11440-66a8-4ea0-a84d-3df64c8a7658 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:59,239 [245] [DEBUG] [app] Starting request: urn:request:40e302a7-54d9-4a44-9406-63403782c9c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:26:59,239 [245] [DEBUG] [app] Ending request: urn:request:40e302a7-54d9-4a44-9406-63403782c9c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40e302a7-54d9-4a44-9406-63403782c9c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:59,240 [246] [DEBUG] [app] Ending request: urn:request:31a11440-66a8-4ea0-a84d-3df64c8a7658 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31a11440-66a8-4ea0-a84d-3df64c8a7658', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:26:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:26:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:26:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:59,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:26:59,243 [237] [DEBUG] [app] Starting request: urn:request:723da1fa-99df-4ab5-a4da-c765f2b6ec8d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:59,243 [238] [DEBUG] [app] Starting request: urn:request:9096e788-7101-49fd-8876-2fb45f8eecf1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:26:59,244 [238] [DEBUG] [app] Ending request: urn:request:9096e788-7101-49fd-8876-2fb45f8eecf1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9096e788-7101-49fd-8876-2fb45f8eecf1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:26:59,244 [237] [DEBUG] [app] Ending request: urn:request:723da1fa-99df-4ab5-a4da-c765f2b6ec8d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:723da1fa-99df-4ab5-a4da-c765f2b6ec8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:26:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:26:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:26:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:26:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:26:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:26:59,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:59,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:26:59,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:59,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:26:59,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:59,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:26:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:26:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:59,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:26:59,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:59,255 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:26:59,257 [236] [DEBUG] [app] Ending request: urn:request:6829b7e8-bc1f-48eb-9b52-b74149a9886d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6829b7e8-bc1f-48eb-9b52-b74149a9886d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:59,257 [235] [DEBUG] [app] Ending request: urn:request:046f4e31-818b-4476-8e68-c1a7d88eacbd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:046f4e31-818b-4476-8e68-c1a7d88eacbd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:26:59,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:26:59,257 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 04:26:59,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:26:59,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:26:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:26:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
exportactionlogsworker stdout | 2025-02-07 04:26:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:26:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:04.936688+00:00 (in 4.998324 seconds)
exportactionlogsworker stdout | 2025-02-07 04:26:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:59 UTC)" (scheduled at 2025-02-07 04:26:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:26:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:26:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:27:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:27:00,177 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:27:01,342 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:27:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:27:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:23.420886+00:00 (in 20.001357 seconds)
repositorygcworker stdout | 2025-02-07 04:27:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:03 UTC)" (scheduled at 2025-02-07 04:27:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:27:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:27:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 3, 419804), True, datetime.datetime(2025, 2, 7, 4, 27, 3, 419804), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:27:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:27:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:27:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:27:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:27:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:33.919286+00:00 (in 29.999518 seconds)
buildlogsarchiver stdout | 2025-02-07 04:27:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:33 UTC)" (scheduled at 2025-02-07 04:27:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:27:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 27, 3, 920013), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:27:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:27:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:27:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:27:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:27:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:59.937899+00:00 (in 55.000776 seconds)
exportactionlogsworker stdout | 2025-02-07 04:27:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:04 UTC)" (scheduled at 2025-02-07 04:27:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:27:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:27:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 4, 937435), True, datetime.datetime(2025, 2, 7, 4, 27, 4, 937435), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:27:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:27:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:27:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:27:05,663 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:27:06,736 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:27:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:18.732147+00:00 (in 9.999489 seconds)
notificationworker stdout | 2025-02-07 04:27:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:18 UTC)" (scheduled at 2025-02-07 04:27:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:27:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:27:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 8, 732976), True, datetime.datetime(2025, 2, 7, 4, 27, 8, 732976), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:27:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:27:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:27:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:27:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:27:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:24.423055+00:00 (in 13.998404 seconds)
gcworker stdout | 2025-02-07 04:27:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:40 UTC)" (scheduled at 2025-02-07 04:27:10.424250+00:00)
gcworker stdout | 2025-02-07 04:27:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:27:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902130430, None, 1, 0])
gcworker stdout | 2025-02-07 04:27:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:27:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:27:10,619 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:27:11,267 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:27:11,708 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:27:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:27:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:41.831521+00:00 (in 29.999508 seconds)
autopruneworker stdout | 2025-02-07 04:27:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:41 UTC)" (scheduled at 2025-02-07 04:27:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:27:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898831836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:27:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:27:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:27:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:27:13,046 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
namespacegcworker stdout | 2025-02-07 04:27:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:27:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:31.621131+00:00 (in 17.996400 seconds)
namespacegcworker stdout | 2025-02-07 04:27:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:32:13 UTC)" (scheduled at 2025-02-07 04:27:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:27:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 27, 13, 625003), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:27:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 27, 13, 625003), True, datetime.datetime(2025, 2, 7, 4, 27, 13, 625003), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:27:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 27, 13, 625003), True, datetime.datetime(2025, 2, 7, 4, 27, 13, 625003), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 27, 13, 625003), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:27:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:27:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:32:13 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:27:13,971 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:27:14,042 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:27:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:27:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:29.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 04:27:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:29 UTC)" (scheduled at 2025-02-07 04:27:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:27:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:27:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:27:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:27:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:27:14,236 [237] [DEBUG] [app] Starting request: urn:request:208f37c7-a9d1-4602-8e90-b92da2386b04 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:14,236 [238] [DEBUG] [app] Starting request: urn:request:edd3a104-f6ff-4014-8d20-49201f5dbc9a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:27:14,239 [245] [DEBUG] [app] Starting request: urn:request:16bf1dfc-7b1a-4034-b452-bca403ebbdd8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:14,239 [239] [DEBUG] [app] Starting request: urn:request:02173039-71d3-460b-b7f5-638618458d89 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:14,239 [245] [DEBUG] [app] Ending request: urn:request:16bf1dfc-7b1a-4034-b452-bca403ebbdd8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:16bf1dfc-7b1a-4034-b452-bca403ebbdd8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:14,239 [239] [DEBUG] [app] Ending request: urn:request:02173039-71d3-460b-b7f5-638618458d89 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02173039-71d3-460b-b7f5-638618458d89', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:27:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:27:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:14,242 [238] [DEBUG] [app] Starting request: urn:request:b8e7d5c0-2b4f-46d1-81bc-16d14dd98337 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:14,242 [238] [DEBUG] [app] Ending request: urn:request:b8e7d5c0-2b4f-46d1-81bc-16d14dd98337 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8e7d5c0-2b4f-46d1-81bc-16d14dd98337', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:27:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:27:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:14,243 [235] [DEBUG] [app] Starting request: urn:request:1059014e-8647-4cf6-88d8-9e0c65641744 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:14,243 [235] [DEBUG] [app] Ending request: urn:request:1059014e-8647-4cf6-88d8-9e0c65641744 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1059014e-8647-4cf6-88d8-9e0c65641744', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:27:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:27:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:14,253 [238] [DEBUG] [app] Ending request: urn:request:edd3a104-f6ff-4014-8d20-49201f5dbc9a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:edd3a104-f6ff-4014-8d20-49201f5dbc9a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:27:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:27:14,255 [237] [DEBUG] [app] Ending request: urn:request:208f37c7-a9d1-4602-8e90-b92da2386b04 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:208f37c7-a9d1-4602-8e90-b92da2386b04', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:27:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
autopruneworker stdout | 2025-02-07 04:27:15,080 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:15,585 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:27:16,079 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:27:16,083 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:27:16,272 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:27:16,664 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:27:16,722 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:27:16,727 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:27:16,730 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:27:16,734 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:27:16,736 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:27:17,106 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:27:17,242 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:27:17,250 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:27:17,741 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:27:17,746 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:27:17,749 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:27:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:28.732147+00:00 (in 9.999497 seconds)
notificationworker stdout | 2025-02-07 04:27:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:28 UTC)" (scheduled at 2025-02-07 04:27:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:27:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:27:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 18, 732936), True, datetime.datetime(2025, 2, 7, 4, 27, 18, 732936), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:27:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:27:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:27:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:27:18,920 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:27:18,924 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:27:18,927 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:27:18,930 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:27:18,937 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:27:18,940 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:27:18,943 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:27:18,947 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:27:18,950 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:27:18,961 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:27:19,037 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:27:19,390 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:27:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:27:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:03.419110+00:00 (in 39.997757 seconds)
repositorygcworker stdout | 2025-02-07 04:27:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:23 UTC)" (scheduled at 2025-02-07 04:27:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:27:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:27:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:27:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:27:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:40.424250+00:00 (in 16.000753 seconds)
gcworker stdout | 2025-02-07 04:27:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:54 UTC)" (scheduled at 2025-02-07 04:27:24.423055+00:00)
gcworker stdout | 2025-02-07 04:27:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:27:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:27:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:27:28,426 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:27:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:33.814166+00:00 (in 5.081520 seconds)
notificationworker stdout | 2025-02-07 04:27:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:38 UTC)" (scheduled at 2025-02-07 04:27:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:27:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:27:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 28, 732965), True, datetime.datetime(2025, 2, 7, 4, 27, 28, 732965), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:27:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:27:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:27:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:27:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:27:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:44.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 04:27:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:44 UTC)" (scheduled at 2025-02-07 04:27:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:27:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:27:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:27:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:27:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:27:29,235 [236] [DEBUG] [app] Starting request: urn:request:afada98c-9985-4c1d-9471-5be1bae411e0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:29,235 [237] [DEBUG] [app] Starting request: urn:request:8e636113-858c-4553-b3dc-7571407e169d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:27:29,238 [246] [DEBUG] [app] Starting request: urn:request:9947be27-21c3-46df-afe5-11a9b35f60a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:29,238 [239] [DEBUG] [app] Starting request: urn:request:a2710c1d-34da-4252-a1cf-f59a4aa020d8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:29,238 [246] [DEBUG] [app] Ending request: urn:request:9947be27-21c3-46df-afe5-11a9b35f60a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9947be27-21c3-46df-afe5-11a9b35f60a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:29,238 [239] [DEBUG] [app] Ending request: urn:request:a2710c1d-34da-4252-a1cf-f59a4aa020d8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2710c1d-34da-4252-a1cf-f59a4aa020d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:27:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:27:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:27:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:29,241 [236] [DEBUG] [app] Starting request: urn:request:88c3b8fe-2746-4597-bfaa-fcaea367140b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:29,241 [236] [DEBUG] [app] Ending request: urn:request:88c3b8fe-2746-4597-bfaa-fcaea367140b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:88c3b8fe-2746-4597-bfaa-fcaea367140b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:27:29,241 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:27:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:29,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:29,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:29,242 [235] [DEBUG] [app] Starting request: urn:request:eacf6902-47af-4492-a62e-f0e1a939ace5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:29,242 [235] [DEBUG] [app] Ending request: urn:request:eacf6902-47af-4492-a62e-f0e1a939ace5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eacf6902-47af-4492-a62e-f0e1a939ace5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:27:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:27:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:29,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:29,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:29,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:29,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:29,253 [236] [DEBUG] [app] Ending request: urn:request:afada98c-9985-4c1d-9471-5be1bae411e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:afada98c-9985-4c1d-9471-5be1bae411e0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:29,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:27:29,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:27:29,253 [237] [DEBUG] [app] Ending request: urn:request:8e636113-858c-4553-b3dc-7571407e169d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8e636113-858c-4553-b3dc-7571407e169d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:27:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:27:30,214 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:27:31,353 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:27:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:27:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:32.625945+00:00 (in 1.004362 seconds)
namespacegcworker stdout | 2025-02-07 04:27:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:31 UTC)" (scheduled at 2025-02-07 04:27:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:27:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:27:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 31, 621881), True, datetime.datetime(2025, 2, 7, 4, 27, 31, 621881), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:27:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:27:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:27:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:27:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:27:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:31.621131+00:00 (in 58.994738 seconds)
namespacegcworker stdout | 2025-02-07 04:27:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:32 UTC)" (scheduled at 2025-02-07 04:27:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:27:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:27:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:27:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:38.732147+00:00 (in 4.917473 seconds)
notificationworker stdout | 2025-02-07 04:27:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:32:33 UTC)" (scheduled at 2025-02-07 04:27:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:27:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 27, 33, 814949), 'notification/%'])
notificationworker stdout | 2025-02-07 04:27:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 27, 33, 814949), True, datetime.datetime(2025, 2, 7, 4, 27, 33, 814949), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:27:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 27, 33, 814949), True, datetime.datetime(2025, 2, 7, 4, 27, 33, 814949), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 27, 33, 814949), 'notification/%'])
notificationworker stdout | 2025-02-07 04:27:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:27:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:32:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:27:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:27:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:03.919286+00:00 (in 29.999571 seconds)
buildlogsarchiver stdout | 2025-02-07 04:27:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:03 UTC)" (scheduled at 2025-02-07 04:27:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:27:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 27, 33, 919977), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:27:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:27:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:27:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:27:35,671 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:27:36,744 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:56.713547+00:00 (in 18.997189 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:37 UTC)" (scheduled at 2025-02-07 04:27:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:27:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:48.732147+00:00 (in 9.999509 seconds)
notificationworker stdout | 2025-02-07 04:27:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:48 UTC)" (scheduled at 2025-02-07 04:27:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:27:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:27:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 38, 732946), True, datetime.datetime(2025, 2, 7, 4, 27, 38, 732946), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:27:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:27:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:27:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:27:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:27:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:54.423055+00:00 (in 13.998363 seconds)
gcworker stdout | 2025-02-07 04:27:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:10 UTC)" (scheduled at 2025-02-07 04:27:40.424250+00:00)
gcworker stdout | 2025-02-07 04:27:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:27:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902160430, None, 1, 0])
gcworker stdout | 2025-02-07 04:27:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:27:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:27:40,655 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:27:41,298 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:27:41,744 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:27:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:27:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:11.831521+00:00 (in 29.999565 seconds)
autopruneworker stdout | 2025-02-07 04:27:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:11 UTC)" (scheduled at 2025-02-07 04:27:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:27:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898861836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:27:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:27:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:27:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:27:43,083 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:27:44,007 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:27:44,078 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:27:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:27:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:59.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 04:27:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:59 UTC)" (scheduled at 2025-02-07 04:27:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:27:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:27:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:27:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:27:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:27:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:27:44,236 [235] [DEBUG] [app] Starting request: urn:request:5e9f583b-d9ab-400a-8e10-cae95be8d53c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:44,236 [238] [DEBUG] [app] Starting request: urn:request:5197e7ae-4fea-49a9-bc6d-cc899b1cfae5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:27:44,239 [239] [DEBUG] [app] Starting request: urn:request:136bdf3d-ed09-476e-a411-bead03b47ae8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:44,239 [246] [DEBUG] [app] Starting request: urn:request:c309807c-a467-43f4-b5df-3e8b38f0d432 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:44,240 [239] [DEBUG] [app] Ending request: urn:request:136bdf3d-ed09-476e-a411-bead03b47ae8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:136bdf3d-ed09-476e-a411-bead03b47ae8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:44,240 [246] [DEBUG] [app] Ending request: urn:request:c309807c-a467-43f4-b5df-3e8b38f0d432 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c309807c-a467-43f4-b5df-3e8b38f0d432', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:27:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:27:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:44,243 [238] [DEBUG] [app] Starting request: urn:request:5a951bd6-f590-4c1a-a183-5352d9815853 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:44,243 [238] [DEBUG] [app] Ending request: urn:request:5a951bd6-f590-4c1a-a183-5352d9815853 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a951bd6-f590-4c1a-a183-5352d9815853', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:27:44,243 [237] [DEBUG] [app] Starting request: urn:request:5473fb19-55f7-4b9a-a0c2-c9dc5497b96a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:27:44,243 [237] [DEBUG] [app] Ending request: urn:request:5473fb19-55f7-4b9a-a0c2-c9dc5497b96a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5473fb19-55f7-4b9a-a0c2-c9dc5497b96a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:27:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:27:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:27:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:44,255 [235] [DEBUG] [app] Ending request: urn:request:5e9f583b-d9ab-400a-8e10-cae95be8d53c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5e9f583b-d9ab-400a-8e10-cae95be8d53c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:44,255 [238] [DEBUG] [app] Ending request: urn:request:5197e7ae-4fea-49a9-bc6d-cc899b1cfae5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5197e7ae-4fea-49a9-bc6d-cc899b1cfae5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:27:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:27:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:27:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
autopruneworker stdout | 2025-02-07 04:27:45,115 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:45,619 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:27:46,115 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:27:46,118 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:27:46,282 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:27:46,701 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:27:46,731 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:27:46,734 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:27:46,738 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:27:46,741 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:27:46,743 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:27:47,133 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:27:47,277 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:27:47,286 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:27:47,748 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:27:47,753 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:27:47,756 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:27:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:50.814677+00:00 (in 2.082016 seconds)
notificationworker stdout | 2025-02-07 04:27:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:58 UTC)" (scheduled at 2025-02-07 04:27:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:27:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:27:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 48, 732970), True, datetime.datetime(2025, 2, 7, 4, 27, 48, 732970), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:27:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:27:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:27:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:27:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:27:48,931 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:27:48,935 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:27:48,937 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:27:48,940 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:27:48,943 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:27:48,947 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:27:48,950 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:27:48,954 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:27:48,957 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:27:48,996 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:27:49,073 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:27:49,401 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:27:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:27:58.732147+00:00 (in 7.916977 seconds)
notificationworker stdout | 2025-02-07 04:27:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:50 UTC)" (scheduled at 2025-02-07 04:27:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:27:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:27:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:27:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:27:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:10.424250+00:00 (in 16.000686 seconds)
gcworker stdout | 2025-02-07 04:27:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:24 UTC)" (scheduled at 2025-02-07 04:27:54.423055+00:00)
gcworker stdout | 2025-02-07 04:27:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:27:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:27:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:27:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:56.118855+00:00 (in 59.999494 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:27:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:56 UTC)" (scheduled at 2025-02-07 04:27:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:27:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:27:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:27:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:37.715878+00:00 (in 41.001833 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:56 UTC)" (scheduled at 2025-02-07 04:27:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:27:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:27:58,457 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:27:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:27:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:08.732147+00:00 (in 9.999524 seconds)
notificationworker stdout | 2025-02-07 04:27:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:08 UTC)" (scheduled at 2025-02-07 04:27:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:27:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:27:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 27, 58, 732906), True, datetime.datetime(2025, 2, 7, 4, 27, 58, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:27:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:27:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:27:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:27:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:27:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:14.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 04:27:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:14 UTC)" (scheduled at 2025-02-07 04:27:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:27:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:27:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:27:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:27:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:27:59,236 [235] [DEBUG] [app] Starting request: urn:request:b27f44fd-946c-4e56-be46-ecebcade1706 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:59,236 [237] [DEBUG] [app] Starting request: urn:request:77d1e7e9-9e77-448c-8bb6-86a726b37c3e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:27:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:27:59,238 [239] [DEBUG] [app] Starting request: urn:request:0513d584-68ff-4908-b549-7e740f14cb28 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:59,238 [245] [DEBUG] [app] Starting request: urn:request:4559f4ab-ff92-4764-8848-60dfb8474c72 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:27:59,239 [245] [DEBUG] [app] Ending request: urn:request:4559f4ab-ff92-4764-8848-60dfb8474c72 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4559f4ab-ff92-4764-8848-60dfb8474c72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:59,239 [239] [DEBUG] [app] Ending request: urn:request:0513d584-68ff-4908-b549-7e740f14cb28 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0513d584-68ff-4908-b549-7e740f14cb28', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:27:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:27:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:27:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:27:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:27:59,242 [237] [DEBUG] [app] Starting request: urn:request:952719a2-67b2-4f7e-bd8f-5d4eb75088f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:59,242 [236] [DEBUG] [app] Starting request: urn:request:ad24cc59-6e3c-429c-81b3-7739e14aee74 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:27:59,242 [237] [DEBUG] [app] Ending request: urn:request:952719a2-67b2-4f7e-bd8f-5d4eb75088f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:952719a2-67b2-4f7e-bd8f-5d4eb75088f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:27:59,242 [236] [DEBUG] [app] Ending request: urn:request:ad24cc59-6e3c-429c-81b3-7739e14aee74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad24cc59-6e3c-429c-81b3-7739e14aee74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:27:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:27:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:27:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:27:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:27:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:27:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:27:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:27:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:27:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:27:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:27:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:27:59,253 [237] [DEBUG] [app] Ending request: urn:request:77d1e7e9-9e77-448c-8bb6-86a726b37c3e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77d1e7e9-9e77-448c-8bb6-86a726b37c3e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:27:59,254 [235] [DEBUG] [app] Ending request: urn:request:b27f44fd-946c-4e56-be46-ecebcade1706 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b27f44fd-946c-4e56-be46-ecebcade1706', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:27:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:27:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:27:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:27:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:27:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:27:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:27:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:04.936688+00:00 (in 4.998303 seconds)
exportactionlogsworker stdout | 2025-02-07 04:27:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:59 UTC)" (scheduled at 2025-02-07 04:27:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:27:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:27:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:28:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:28:00,226 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:28:01,386 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:28:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:28:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:23.420886+00:00 (in 20.001284 seconds)
repositorygcworker stdout | 2025-02-07 04:28:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:03 UTC)" (scheduled at 2025-02-07 04:28:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:28:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:28:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 3, 419915), True, datetime.datetime(2025, 2, 7, 4, 28, 3, 419915), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:28:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:28:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:28:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:28:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:28:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:33.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 04:28:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:33 UTC)" (scheduled at 2025-02-07 04:28:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:28:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 28, 3, 919978), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:28:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:28:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:28:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:28:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:28:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:59.937899+00:00 (in 55.000805 seconds)
exportactionlogsworker stdout | 2025-02-07 04:28:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:04 UTC)" (scheduled at 2025-02-07 04:28:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:28:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:28:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 4, 937382), True, datetime.datetime(2025, 2, 7, 4, 28, 4, 937382), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:28:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:28:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:28:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:28:05,689 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:28:06,774 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:28:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:28:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:18.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 04:28:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:18 UTC)" (scheduled at 2025-02-07 04:28:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:28:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:28:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 8, 732864), True, datetime.datetime(2025, 2, 7, 4, 28, 8, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:28:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:28:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:28:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:28:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:28:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:24.423055+00:00 (in 13.998272 seconds)
gcworker stdout | 2025-02-07 04:28:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:40 UTC)" (scheduled at 2025-02-07 04:28:10.424250+00:00)
gcworker stdout | 2025-02-07 04:28:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:28:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902190431, None, 1, 0])
gcworker stdout | 2025-02-07 04:28:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:28:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:28:10,666 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:28:11,331 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:28:11,781 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:28:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:28:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:41.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 04:28:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:41 UTC)" (scheduled at 2025-02-07 04:28:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:28:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898891836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:28:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:28:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:28:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:28:13,119 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:28:14,017 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:28:14,088 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:28:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:28:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:29.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:28:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:29 UTC)" (scheduled at 2025-02-07 04:28:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:28:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:28:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:28:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:28:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:28:14,235 [237] [DEBUG] [app] Starting request: urn:request:49c77e72-3c01-4d26-b4fe-ff6976b01aac (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:14,235 [235] [DEBUG] [app] Starting request: urn:request:c9fbebf0-deef-47e3-8999-e5271d2c8576 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:28:14,238 [246] [DEBUG] [app] Starting request: urn:request:2cd2922d-ef92-4fa1-bd3d-002600e51f4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:14,238 [239] [DEBUG] [app] Starting request: urn:request:bbc9dc11-0a77-402d-b433-824d937be831 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:14,238 [246] [DEBUG] [app] Ending request: urn:request:2cd2922d-ef92-4fa1-bd3d-002600e51f4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2cd2922d-ef92-4fa1-bd3d-002600e51f4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:28:14,238 [239] [DEBUG] [app] Ending request: urn:request:bbc9dc11-0a77-402d-b433-824d937be831 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bbc9dc11-0a77-402d-b433-824d937be831', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:28:14,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:28:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:14,238 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:28:14,239 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:14,240 [237] [DEBUG] [app] Starting request: urn:request:738b6b64-24b2-49e9-b101-f0ded3930562 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:14,241 [237] [DEBUG] [app] Ending request: urn:request:738b6b64-24b2-49e9-b101-f0ded3930562 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:738b6b64-24b2-49e9-b101-f0ded3930562', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:28:14,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:28:14,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:14,241 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:14,242 [238] [DEBUG] [app] Starting request: urn:request:b8afb0f4-e1f8-4929-acd6-a780493dc124 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:14,243 [238] [DEBUG] [app] Ending request: urn:request:b8afb0f4-e1f8-4929-acd6-a780493dc124 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8afb0f4-e1f8-4929-acd6-a780493dc124', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:28:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:14,245 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:14,245 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:14,253 [235] [DEBUG] [app] Ending request: urn:request:c9fbebf0-deef-47e3-8999-e5271d2c8576 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9fbebf0-deef-47e3-8999-e5271d2c8576', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:28:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:28:14,254 [237] [DEBUG] [app] Ending request: urn:request:49c77e72-3c01-4d26-b4fe-ff6976b01aac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:49c77e72-3c01-4d26-b4fe-ff6976b01aac', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:28:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
autopruneworker stdout | 2025-02-07 04:28:15,129 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:15,656 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:28:16,138 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:28:16,155 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:28:16,298 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:28:16,734 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-web stdout | 2025-02-07 04:28:16,739 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:28:16,742 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:28:16,745 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:28:16,748 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:28:16,751 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
gcworker stdout | 2025-02-07 04:28:17,143 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:28:17,294 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:28:17,317 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:28:17,756 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:28:17,761 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:28:17,764 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:28:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:28:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:28.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 04:28:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:28 UTC)" (scheduled at 2025-02-07 04:28:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:28:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:28:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 18, 732865), True, datetime.datetime(2025, 2, 7, 4, 28, 18, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:28:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:28:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:28:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:28:18,943 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:28:18,946 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:28:18,948 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:28:18,952 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:28:18,955 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:28:18,958 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:28:18,960 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:28:18,964 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:28:18,969 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:28:19,017 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:28:19,088 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:28:19,434 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:28:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:28:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:32.420707+00:00 (in 8.999262 seconds)
repositorygcworker stdout | 2025-02-07 04:28:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:23 UTC)" (scheduled at 2025-02-07 04:28:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:28:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:28:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:28:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:28:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 04:28:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:54 UTC)" (scheduled at 2025-02-07 04:28:24.423055+00:00)
gcworker stdout | 2025-02-07 04:28:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:28:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:28:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:28:28,469 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:28:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:28:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:38.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:28:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:38 UTC)" (scheduled at 2025-02-07 04:28:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:28:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:28:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 28, 732868), True, datetime.datetime(2025, 2, 7, 4, 28, 28, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:28:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:28:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:28:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:28:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:28:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:44.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 04:28:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:44 UTC)" (scheduled at 2025-02-07 04:28:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:28:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:28:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:28:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:28:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:28:29,235 [236] [DEBUG] [app] Starting request: urn:request:f1c60940-bc65-434f-b231-5ad4033af88e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:29,235 [235] [DEBUG] [app] Starting request: urn:request:8d666b1f-ddb1-46a2-be7b-21db39967a32 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:28:29,238 [246] [DEBUG] [app] Starting request: urn:request:a7301742-b2ba-4728-b881-95b6392ca280 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:29,238 [245] [DEBUG] [app] Starting request: urn:request:34e0f869-8471-4453-a11e-1ebfdcb689bc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:29,239 [245] [DEBUG] [app] Ending request: urn:request:34e0f869-8471-4453-a11e-1ebfdcb689bc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34e0f869-8471-4453-a11e-1ebfdcb689bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:28:29,239 [246] [DEBUG] [app] Ending request: urn:request:a7301742-b2ba-4728-b881-95b6392ca280 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a7301742-b2ba-4728-b881-95b6392ca280', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:28:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:28:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:29,242 [236] [DEBUG] [app] Starting request: urn:request:2944a0e2-a2a8-4bb7-a8ff-744d138a79fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:29,242 [238] [DEBUG] [app] Starting request: urn:request:90220a9d-3803-4324-9e86-1d15b778d6f3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:29,242 [236] [DEBUG] [app] Ending request: urn:request:2944a0e2-a2a8-4bb7-a8ff-744d138a79fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2944a0e2-a2a8-4bb7-a8ff-744d138a79fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:28:29,242 [238] [DEBUG] [app] Ending request: urn:request:90220a9d-3803-4324-9e86-1d15b778d6f3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:90220a9d-3803-4324-9e86-1d15b778d6f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:28:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:28:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:29,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:29,253 [236] [DEBUG] [app] Ending request: urn:request:f1c60940-bc65-434f-b231-5ad4033af88e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1c60940-bc65-434f-b231-5ad4033af88e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:29,253 [235] [DEBUG] [app] Ending request: urn:request:8d666b1f-ddb1-46a2-be7b-21db39967a32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d666b1f-ddb1-46a2-be7b-21db39967a32', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:29,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:28:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:28:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:28:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:28:30,249 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:28:31,401 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:28:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:28:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:32.625945+00:00 (in 1.004335 seconds)
namespacegcworker stdout | 2025-02-07 04:28:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:31 UTC)" (scheduled at 2025-02-07 04:28:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:28:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:28:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 31, 621884), True, datetime.datetime(2025, 2, 7, 4, 28, 31, 621884), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:28:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:28:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:28:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:28:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:28:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:03.419110+00:00 (in 30.997961 seconds)
repositorygcworker stdout | 2025-02-07 04:28:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:33:32 UTC)" (scheduled at 2025-02-07 04:28:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:28:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 28, 32, 421418), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:28:32,427 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 28, 32, 421418), True, datetime.datetime(2025, 2, 7, 4, 28, 32, 421418), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:28:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 28, 32, 421418), True, datetime.datetime(2025, 2, 7, 4, 28, 32, 421418), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 28, 32, 421418), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:28:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:28:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:33:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:28:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:28:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:31.621131+00:00 (in 58.994730 seconds)
namespacegcworker stdout | 2025-02-07 04:28:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:32 UTC)" (scheduled at 2025-02-07 04:28:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:28:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:28:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:28:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:28:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:03.919286+00:00 (in 29.999532 seconds)
buildlogsarchiver stdout | 2025-02-07 04:28:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:03 UTC)" (scheduled at 2025-02-07 04:28:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:28:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 28, 33, 920042), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:28:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:28:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:28:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:28:35,710 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:28:36,796 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:56.713547+00:00 (in 18.997177 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:37 UTC)" (scheduled at 2025-02-07 04:28:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:28:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:28:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:48.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 04:28:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:48 UTC)" (scheduled at 2025-02-07 04:28:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:28:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:28:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 38, 732865), True, datetime.datetime(2025, 2, 7, 4, 28, 38, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:28:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:28:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:28:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:28:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:28:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:54.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 04:28:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:10 UTC)" (scheduled at 2025-02-07 04:28:40.424250+00:00)
gcworker stdout | 2025-02-07 04:28:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:28:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902220431, None, 1, 0])
gcworker stdout | 2025-02-07 04:28:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:28:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:28:40,702 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:28:41,368 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:28:41,794 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:28:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:28:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:11.831521+00:00 (in 29.999557 seconds)
autopruneworker stdout | 2025-02-07 04:28:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:11 UTC)" (scheduled at 2025-02-07 04:28:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:28:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898921836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:28:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:28:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:28:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:11 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:28:43,129 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:28:44,054 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:28:44,118 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:28:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:28:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:59.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 04:28:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:59 UTC)" (scheduled at 2025-02-07 04:28:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:28:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:28:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:28:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:28:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:28:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:28:44,235 [235] [DEBUG] [app] Starting request: urn:request:2cba11fb-cc99-4280-b75b-b61712b30aa4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:44,235 [237] [DEBUG] [app] Starting request: urn:request:67ac520a-1503-4577-b281-d8678ceeacf9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:28:44,238 [246] [DEBUG] [app] Starting request: urn:request:6565a173-5b47-4d44-bd21-a9a01a22f08f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:44,238 [239] [DEBUG] [app] Starting request: urn:request:92a2326e-ced6-42ed-8bc5-d41fd8ab4ee1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:44,238 [239] [DEBUG] [app] Ending request: urn:request:92a2326e-ced6-42ed-8bc5-d41fd8ab4ee1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:92a2326e-ced6-42ed-8bc5-d41fd8ab4ee1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:28:44,238 [246] [DEBUG] [app] Ending request: urn:request:6565a173-5b47-4d44-bd21-a9a01a22f08f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6565a173-5b47-4d44-bd21-a9a01a22f08f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:28:44,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:28:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:28:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:44,241 [237] [DEBUG] [app] Starting request: urn:request:50bf1c66-5340-416a-a0ba-c008af4bd078 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:44,241 [238] [DEBUG] [app] Starting request: urn:request:4d6c7506-e39e-4257-8b67-74f7f1ead4a2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:44,241 [237] [DEBUG] [app] Ending request: urn:request:50bf1c66-5340-416a-a0ba-c008af4bd078 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50bf1c66-5340-416a-a0ba-c008af4bd078', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:28:44,241 [238] [DEBUG] [app] Ending request: urn:request:4d6c7506-e39e-4257-8b67-74f7f1ead4a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d6c7506-e39e-4257-8b67-74f7f1ead4a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:28:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:44,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:44,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:44,253 [237] [DEBUG] [app] Ending request: urn:request:67ac520a-1503-4577-b281-d8678ceeacf9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:67ac520a-1503-4577-b281-d8678ceeacf9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:28:44,253 [235] [DEBUG] [app] Ending request: urn:request:2cba11fb-cc99-4280-b75b-b61712b30aa4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2cba11fb-cc99-4280-b75b-b61712b30aa4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:28:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:28:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
autopruneworker stdout | 2025-02-07 04:28:45,141 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:45,692 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:28:46,174 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:28:46,191 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:28:46,331 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:28:46,747 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:28:46,750 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:28:46,753 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:28:46,755 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:28:46,758 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:28:46,768 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:28:47,178 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:28:47,330 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:28:47,352 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:28:47,764 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:28:47,768 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:28:47,771 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:28:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:28:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:50.814677+00:00 (in 2.082101 seconds)
notificationworker stdout | 2025-02-07 04:28:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:58 UTC)" (scheduled at 2025-02-07 04:28:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:28:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:28:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 48, 732857), True, datetime.datetime(2025, 2, 7, 4, 28, 48, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:28:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:28:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:28:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:28:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:28:48,953 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:28:48,956 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:28:48,959 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:28:48,962 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:28:48,965 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:28:48,968 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:28:48,971 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:28:48,975 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:28:48,979 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:28:49,054 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:28:49,124 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:28:49,470 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:28:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:28:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:28:58.732147+00:00 (in 7.917021 seconds)
notificationworker stdout | 2025-02-07 04:28:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:50 UTC)" (scheduled at 2025-02-07 04:28:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:28:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:28:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:28:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:28:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:10.424250+00:00 (in 16.000748 seconds)
gcworker stdout | 2025-02-07 04:28:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:24 UTC)" (scheduled at 2025-02-07 04:28:54.423055+00:00)
gcworker stdout | 2025-02-07 04:28:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:28:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:28:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:28:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:56.118855+00:00 (in 59.999460 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:28:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:56 UTC)" (scheduled at 2025-02-07 04:28:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:28:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:28:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:28:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:37.715878+00:00 (in 41.001819 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:56 UTC)" (scheduled at 2025-02-07 04:28:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:28:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:28:58,505 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:28:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:28:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:08.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 04:28:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:08 UTC)" (scheduled at 2025-02-07 04:28:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:28:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:28:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 28, 58, 732857), True, datetime.datetime(2025, 2, 7, 4, 28, 58, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:28:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:28:58,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:28:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:28:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:28:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:14.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 04:28:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:14 UTC)" (scheduled at 2025-02-07 04:28:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:28:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:28:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:28:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:28:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:28:59,235 [235] [DEBUG] [app] Starting request: urn:request:00659b24-4696-41e1-bb51-91ca93eb385c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:59,235 [238] [DEBUG] [app] Starting request: urn:request:14edb054-bf9e-42ae-834f-18e77bb81d54 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:28:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:28:59,238 [239] [DEBUG] [app] Starting request: urn:request:1cf72ea3-b525-440c-8198-a727c560ee4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:59,238 [245] [DEBUG] [app] Starting request: urn:request:5dfe4ad2-aa4d-4c6e-af85-f012d62d8af6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:28:59,239 [239] [DEBUG] [app] Ending request: urn:request:1cf72ea3-b525-440c-8198-a727c560ee4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1cf72ea3-b525-440c-8198-a727c560ee4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:28:59,239 [245] [DEBUG] [app] Ending request: urn:request:5dfe4ad2-aa4d-4c6e-af85-f012d62d8af6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5dfe4ad2-aa4d-4c6e-af85-f012d62d8af6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:28:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:28:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:28:59,243 [236] [DEBUG] [app] Starting request: urn:request:6383731f-5001-4761-bbcd-322383d55c4a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:59,243 [237] [DEBUG] [app] Starting request: urn:request:c708e526-d42b-4cd0-b0bd-705be01aeb1e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:28:59,243 [237] [DEBUG] [app] Ending request: urn:request:c708e526-d42b-4cd0-b0bd-705be01aeb1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c708e526-d42b-4cd0-b0bd-705be01aeb1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:28:59,243 [236] [DEBUG] [app] Ending request: urn:request:6383731f-5001-4761-bbcd-322383d55c4a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6383731f-5001-4761-bbcd-322383d55c4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:28:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:28:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:28:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:28:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:28:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:28:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:28:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:28:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:28:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:28:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:28:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:28:59,255 [238] [DEBUG] [app] Ending request: urn:request:14edb054-bf9e-42ae-834f-18e77bb81d54 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14edb054-bf9e-42ae-834f-18e77bb81d54', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:59,255 [235] [DEBUG] [app] Ending request: urn:request:00659b24-4696-41e1-bb51-91ca93eb385c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:00659b24-4696-41e1-bb51-91ca93eb385c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:28:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:28:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:28:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 04:28:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:28:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:28:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:28:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:28:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:04.936688+00:00 (in 4.998315 seconds)
exportactionlogsworker stdout | 2025-02-07 04:28:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:59 UTC)" (scheduled at 2025-02-07 04:28:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:28:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:28:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:29:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:29:00,257 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:29:01,430 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:29:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:29:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:23.420886+00:00 (in 20.001291 seconds)
repositorygcworker stdout | 2025-02-07 04:29:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:03 UTC)" (scheduled at 2025-02-07 04:29:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:29:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:29:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 3, 419870), True, datetime.datetime(2025, 2, 7, 4, 29, 3, 419870), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:29:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:29:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:29:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:29:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:29:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:33.919286+00:00 (in 29.999575 seconds)
buildlogsarchiver stdout | 2025-02-07 04:29:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:33 UTC)" (scheduled at 2025-02-07 04:29:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:29:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 29, 3, 919982), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:29:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:29:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:29:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:29:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:29:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:52.937648+00:00 (in 48.000545 seconds)
exportactionlogsworker stdout | 2025-02-07 04:29:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:04 UTC)" (scheduled at 2025-02-07 04:29:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:29:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:29:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 4, 937395), True, datetime.datetime(2025, 2, 7, 4, 29, 4, 937395), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:29:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:29:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:29:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:29:05,729 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:29:06,811 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:29:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:29:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:18.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:29:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:18 UTC)" (scheduled at 2025-02-07 04:29:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:29:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:29:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 8, 732857), True, datetime.datetime(2025, 2, 7, 4, 29, 8, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:29:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:29:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:29:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:29:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:29:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:24.423055+00:00 (in 13.998372 seconds)
gcworker stdout | 2025-02-07 04:29:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:40 UTC)" (scheduled at 2025-02-07 04:29:10.424250+00:00)
gcworker stdout | 2025-02-07 04:29:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:29:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902250430, None, 1, 0])
gcworker stdout | 2025-02-07 04:29:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:29:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:29:10,739 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:29:11,401 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:29:11,826 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
autopruneworker stdout | 2025-02-07 04:29:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:29:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:41.831521+00:00 (in 29.999565 seconds)
autopruneworker stdout | 2025-02-07 04:29:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:41 UTC)" (scheduled at 2025-02-07 04:29:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:29:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898951836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:29:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:29:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:29:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:41 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:29:13,166 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:29:14,090 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:29:14,134 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:29:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:29:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:29.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:29:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:29 UTC)" (scheduled at 2025-02-07 04:29:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:29:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:29:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:29:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:29:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:29:14,235 [235] [DEBUG] [app] Starting request: urn:request:cf95ae3d-0025-4964-b789-8b4d068f9c77 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:14,235 [238] [DEBUG] [app] Starting request: urn:request:45e9a901-8890-4b12-a6b3-5d6eb79db6a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:29:14,238 [246] [DEBUG] [app] Starting request: urn:request:c9528a19-74cb-4f85-8fd9-b288239518da (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:14,239 [246] [DEBUG] [app] Ending request: urn:request:c9528a19-74cb-4f85-8fd9-b288239518da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c9528a19-74cb-4f85-8fd9-b288239518da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:29:14,239 [245] [DEBUG] [app] Starting request: urn:request:59a3189b-caa2-41eb-b544-c8f2649a6c44 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:29:14,239 [245] [DEBUG] [app] Ending request: urn:request:59a3189b-caa2-41eb-b544-c8f2649a6c44 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59a3189b-caa2-41eb-b544-c8f2649a6c44', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:29:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:29:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:29:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:14,241 [236] [DEBUG] [app] Starting request: urn:request:2519b67e-41c8-4cb9-93bd-e2bfeb419f33 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:14,242 [236] [DEBUG] [app] Ending request: urn:request:2519b67e-41c8-4cb9-93bd-e2bfeb419f33 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2519b67e-41c8-4cb9-93bd-e2bfeb419f33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:29:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:29:14,242 [235] [DEBUG] [app] Starting request: urn:request:4e880313-045f-4003-8dab-d381c1e68fd2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:14,243 [235] [DEBUG] [app] Ending request: urn:request:4e880313-045f-4003-8dab-d381c1e68fd2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4e880313-045f-4003-8dab-d381c1e68fd2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:29:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:29:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:14,254 [238] [DEBUG] [app] Ending request: urn:request:45e9a901-8890-4b12-a6b3-5d6eb79db6a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:45e9a901-8890-4b12-a6b3-5d6eb79db6a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:14,254 [235] [DEBUG] [app] Ending request: urn:request:cf95ae3d-0025-4964-b789-8b4d068f9c77 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf95ae3d-0025-4964-b789-8b4d068f9c77', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:29:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:29:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:29:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
autopruneworker stdout | 2025-02-07 04:29:15,177 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:15,707 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:29:16,202 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:29:16,228 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:29:16,341 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:29:16,755 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:29:16,758 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:29:16,761 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:29:16,765 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:29:16,767 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:29:16,799 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:29:17,216 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:29:17,363 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:29:17,388 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:29:17,773 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:29:17,776 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:29:17,779 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:29:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:29:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:28.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 04:29:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:28 UTC)" (scheduled at 2025-02-07 04:29:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:29:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:29:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 18, 732855), True, datetime.datetime(2025, 2, 7, 4, 29, 18, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:29:18,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:29:18,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:29:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:29:18,962 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:29:18,967 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:29:18,969 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:29:18,974 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:29:18,976 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:29:18,979 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:29:18,981 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:29:18,985 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:29:18,988 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:29:19,090 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:29:19,152 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:29:19,507 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:29:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:29:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:03.419110+00:00 (in 39.997752 seconds)
repositorygcworker stdout | 2025-02-07 04:29:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:23 UTC)" (scheduled at 2025-02-07 04:29:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:29:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:29:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:29:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:29:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:40.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 04:29:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:54 UTC)" (scheduled at 2025-02-07 04:29:24.423055+00:00)
gcworker stdout | 2025-02-07 04:29:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:29:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:29:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:29:28,512 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:29:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:29:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:38.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:29:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:38 UTC)" (scheduled at 2025-02-07 04:29:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:29:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:29:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 28, 732870), True, datetime.datetime(2025, 2, 7, 4, 29, 28, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:29:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:29:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:29:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:29:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:29:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:44.225378+00:00 (in 14.999531 seconds)
quotatotalworker stdout | 2025-02-07 04:29:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:44 UTC)" (scheduled at 2025-02-07 04:29:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:29:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:29:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:29:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:29:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:29:29,235 [235] [DEBUG] [app] Starting request: urn:request:48418063-dd34-4744-8add-379a89941997 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:29,235 [238] [DEBUG] [app] Starting request: urn:request:5f646419-da80-40dd-98c0-9577570b4d83 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:29:29,238 [239] [DEBUG] [app] Starting request: urn:request:4c77e4f9-c021-4573-add8-470503c10374 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:29,238 [245] [DEBUG] [app] Starting request: urn:request:ed6bfb37-8e6c-4cf3-8f37-7ef0451ccb88 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:29,238 [239] [DEBUG] [app] Ending request: urn:request:4c77e4f9-c021-4573-add8-470503c10374 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c77e4f9-c021-4573-add8-470503c10374', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:29:29,238 [245] [DEBUG] [app] Ending request: urn:request:ed6bfb37-8e6c-4cf3-8f37-7ef0451ccb88 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ed6bfb37-8e6c-4cf3-8f37-7ef0451ccb88', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:29:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:29:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:29:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:29,241 [237] [DEBUG] [app] Starting request: urn:request:31a0bd0a-445c-4a39-9f17-afcbbe14d924 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:29,242 [235] [DEBUG] [app] Starting request: urn:request:5bb79afd-f56f-4864-9ad2-73ebf4a36742 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:29,242 [237] [DEBUG] [app] Ending request: urn:request:31a0bd0a-445c-4a39-9f17-afcbbe14d924 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31a0bd0a-445c-4a39-9f17-afcbbe14d924', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:29:29,242 [235] [DEBUG] [app] Ending request: urn:request:5bb79afd-f56f-4864-9ad2-73ebf4a36742 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5bb79afd-f56f-4864-9ad2-73ebf4a36742', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:29:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:29:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:29:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:29:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:29,253 [238] [DEBUG] [app] Ending request: urn:request:5f646419-da80-40dd-98c0-9577570b4d83 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5f646419-da80-40dd-98c0-9577570b4d83', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:29,253 [235] [DEBUG] [app] Ending request: urn:request:48418063-dd34-4744-8add-379a89941997 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:48418063-dd34-4744-8add-379a89941997', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:29:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:29:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:29:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:29:30,270 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:29:31,467 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:29:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:29:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:32.625945+00:00 (in 1.004357 seconds)
namespacegcworker stdout | 2025-02-07 04:29:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:31 UTC)" (scheduled at 2025-02-07 04:29:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:29:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:29:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 31, 621878), True, datetime.datetime(2025, 2, 7, 4, 29, 31, 621878), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:29:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:29:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:29:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:29:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:29:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:31.621131+00:00 (in 58.994744 seconds)
namespacegcworker stdout | 2025-02-07 04:29:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:32 UTC)" (scheduled at 2025-02-07 04:29:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:29:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:29:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:29:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:29:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:03.919286+00:00 (in 29.999553 seconds)
buildlogsarchiver stdout | 2025-02-07 04:29:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:03 UTC)" (scheduled at 2025-02-07 04:29:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:29:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 29, 33, 919985), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:29:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:29:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:29:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:29:35,749 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:29:36,846 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:56.713547+00:00 (in 18.997208 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:37 UTC)" (scheduled at 2025-02-07 04:29:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:29:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:29:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:48.732147+00:00 (in 9.999397 seconds)
notificationworker stdout | 2025-02-07 04:29:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:48 UTC)" (scheduled at 2025-02-07 04:29:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:29:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:29:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 38, 733125), True, datetime.datetime(2025, 2, 7, 4, 29, 38, 733125), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:29:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:29:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:29:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:29:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:29:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:54.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 04:29:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:10 UTC)" (scheduled at 2025-02-07 04:29:40.424250+00:00)
gcworker stdout | 2025-02-07 04:29:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:29:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902280431, None, 1, 0])
gcworker stdout | 2025-02-07 04:29:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:29:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:29:40,751 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:29:41,437 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:29:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:29:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:11.831521+00:00 (in 29.999527 seconds)
autopruneworker stdout | 2025-02-07 04:29:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:11 UTC)" (scheduled at 2025-02-07 04:29:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:29:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738898981836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:29:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:29:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:29:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:11 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:29:41,862 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:29:43,202 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:29:44,127 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:29:44,171 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:29:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:29:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:59.225378+00:00 (in 14.999514 seconds)
quotatotalworker stdout | 2025-02-07 04:29:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:59 UTC)" (scheduled at 2025-02-07 04:29:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:29:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:29:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:29:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:29:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:29:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:29:44,235 [236] [DEBUG] [app] Starting request: urn:request:5923328b-8b12-4caa-b16d-431a56bbab30 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:44,235 [238] [DEBUG] [app] Starting request: urn:request:79753cc3-02df-45f8-8baa-748638af0f70 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:29:44,238 [246] [DEBUG] [app] Starting request: urn:request:68abd5e7-c21c-46fd-9787-721418bb76a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:44,238 [245] [DEBUG] [app] Starting request: urn:request:e64d41ce-67a8-4281-8259-5a734b05046a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:44,239 [246] [DEBUG] [app] Ending request: urn:request:68abd5e7-c21c-46fd-9787-721418bb76a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68abd5e7-c21c-46fd-9787-721418bb76a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:29:44,239 [245] [DEBUG] [app] Ending request: urn:request:e64d41ce-67a8-4281-8259-5a734b05046a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e64d41ce-67a8-4281-8259-5a734b05046a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:29:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:29:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:29:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:44,242 [238] [DEBUG] [app] Starting request: urn:request:cd47b9b6-ffe3-4f7d-ad64-86db52e27eca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:44,242 [235] [DEBUG] [app] Starting request: urn:request:550875e4-7df0-4e2b-a1d3-31ddaa3092d5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:44,242 [235] [DEBUG] [app] Ending request: urn:request:550875e4-7df0-4e2b-a1d3-31ddaa3092d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:550875e4-7df0-4e2b-a1d3-31ddaa3092d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:29:44,242 [238] [DEBUG] [app] Ending request: urn:request:cd47b9b6-ffe3-4f7d-ad64-86db52e27eca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cd47b9b6-ffe3-4f7d-ad64-86db52e27eca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:29:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:29:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:29:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:44,254 [238] [DEBUG] [app] Ending request: urn:request:79753cc3-02df-45f8-8baa-748638af0f70 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79753cc3-02df-45f8-8baa-748638af0f70', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:44,254 [236] [DEBUG] [app] Ending request: urn:request:5923328b-8b12-4caa-b16d-431a56bbab30 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5923328b-8b12-4caa-b16d-431a56bbab30', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:29:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:29:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:29:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
autopruneworker stdout | 2025-02-07 04:29:45,212 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:45,738 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:29:46,213 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:29:46,263 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:29:46,378 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:29:46,764 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:29:46,768 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:29:46,771 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:29:46,773 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:29:46,776 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:29:46,817 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:29:47,253 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:29:47,387 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:29:47,424 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:29:47,781 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:29:47,784 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:29:47,787 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:29:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:29:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:50.814677+00:00 (in 2.082029 seconds)
notificationworker stdout | 2025-02-07 04:29:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:58 UTC)" (scheduled at 2025-02-07 04:29:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:29:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:29:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 48, 732961), True, datetime.datetime(2025, 2, 7, 4, 29, 48, 732961), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:29:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:29:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:29:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:29:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:29:48,971 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:29:48,979 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:29:48,982 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:29:48,985 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:29:48,988 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:29:48,993 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:29:48,996 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:29:48,998 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:29:49,001 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:29:49,122 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:29:49,163 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:29:49,537 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:29:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:29:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:58.732147+00:00 (in 7.916942 seconds)
notificationworker stdout | 2025-02-07 04:29:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:50 UTC)" (scheduled at 2025-02-07 04:29:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:29:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:29:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:29:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:29:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:29:59.937899+00:00 (in 6.999820 seconds)
exportactionlogsworker stdout | 2025-02-07 04:29:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:34:52 UTC)" (scheduled at 2025-02-07 04:29:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:29:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 29, 52, 938365), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:29:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 29, 52, 938365), True, datetime.datetime(2025, 2, 7, 4, 29, 52, 938365), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:29:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 29, 52, 938365), True, datetime.datetime(2025, 2, 7, 4, 29, 52, 938365), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 29, 52, 938365), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:29:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:29:52,949 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:34:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:29:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:29:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:10.424250+00:00 (in 16.000761 seconds)
gcworker stdout | 2025-02-07 04:29:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:24 UTC)" (scheduled at 2025-02-07 04:29:54.423055+00:00)
gcworker stdout | 2025-02-07 04:29:54,423 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 04:29:54,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:29:54,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:29:54,429 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:29:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:29:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:56.118855+00:00 (in 59.999482 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:29:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:56 UTC)" (scheduled at 2025-02-07 04:29:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:29:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:29:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:29:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:37.715878+00:00 (in 41.001863 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:56 UTC)" (scheduled at 2025-02-07 04:29:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:29:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:29:58,534 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:29:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:29:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:08.732147+00:00 (in 9.999455 seconds)
notificationworker stdout | 2025-02-07 04:29:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:08 UTC)" (scheduled at 2025-02-07 04:29:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:29:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:29:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 29, 58, 732995), True, datetime.datetime(2025, 2, 7, 4, 29, 58, 732995), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:29:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:29:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:29:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:29:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:29:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:14.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 04:29:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:14 UTC)" (scheduled at 2025-02-07 04:29:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:29:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:29:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:29:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:29:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:29:59,235 [237] [DEBUG] [app] Starting request: urn:request:c2747c94-9850-4be7-969c-d6321d499046 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:59,235 [238] [DEBUG] [app] Starting request: urn:request:4c63d56a-cc2b-4015-802b-f9fb150ee278 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:29:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:29:59,238 [246] [DEBUG] [app] Starting request: urn:request:babaf46b-c5f2-4f54-9f78-825b24c14d4a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:59,238 [239] [DEBUG] [app] Starting request: urn:request:c3c137a9-ae9a-4fc8-b413-0db8253ed5c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:29:59,238 [246] [DEBUG] [app] Ending request: urn:request:babaf46b-c5f2-4f54-9f78-825b24c14d4a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:babaf46b-c5f2-4f54-9f78-825b24c14d4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:29:59,239 [239] [DEBUG] [app] Ending request: urn:request:c3c137a9-ae9a-4fc8-b413-0db8253ed5c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3c137a9-ae9a-4fc8-b413-0db8253ed5c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:29:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:29:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:29:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:29:59,242 [236] [DEBUG] [app] Starting request: urn:request:e6ba7079-b6b3-471b-a46b-66df477f18c5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:59,242 [235] [DEBUG] [app] Starting request: urn:request:b7e6bdba-d379-41e7-8d26-0e9066268b47 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:29:59,243 [236] [DEBUG] [app] Ending request: urn:request:e6ba7079-b6b3-471b-a46b-66df477f18c5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6ba7079-b6b3-471b-a46b-66df477f18c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:29:59,243 [235] [DEBUG] [app] Ending request: urn:request:b7e6bdba-d379-41e7-8d26-0e9066268b47 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7e6bdba-d379-41e7-8d26-0e9066268b47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:29:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:29:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:29:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:29:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:29:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:29:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:29:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:29:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:29:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:29:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:29:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:29:59,255 [237] [DEBUG] [app] Ending request: urn:request:c2747c94-9850-4be7-969c-d6321d499046 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2747c94-9850-4be7-969c-d6321d499046', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:29:59,255 [238] [DEBUG] [app] Ending request: urn:request:4c63d56a-cc2b-4015-802b-f9fb150ee278 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c63d56a-cc2b-4015-802b-f9fb150ee278', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:29:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:29:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:29:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:29:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:29:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 04:29:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:29:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:04.936688+00:00 (in 4.998290 seconds)
exportactionlogsworker stdout | 2025-02-07 04:29:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:59 UTC)" (scheduled at 2025-02-07 04:29:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:29:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:29:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:30:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:30:00,291 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:30:01,496 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:30:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:30:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:23.420886+00:00 (in 20.001306 seconds)
repositorygcworker stdout | 2025-02-07 04:30:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:03 UTC)" (scheduled at 2025-02-07 04:30:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:30:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:30:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 3, 419858), True, datetime.datetime(2025, 2, 7, 4, 30, 3, 419858), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:30:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:30:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:30:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:30:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:30:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:33.919286+00:00 (in 29.999517 seconds)
buildlogsarchiver stdout | 2025-02-07 04:30:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:33 UTC)" (scheduled at 2025-02-07 04:30:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:30:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 30, 3, 920038), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:30:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:30:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:30:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:30:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:30:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:59.937899+00:00 (in 55.000731 seconds)
exportactionlogsworker stdout | 2025-02-07 04:30:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:04 UTC)" (scheduled at 2025-02-07 04:30:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:30:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:30:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 4, 937498), True, datetime.datetime(2025, 2, 7, 4, 30, 4, 937498), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:30:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:30:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:30:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:30:05,774 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:30:06,860 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:30:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:30:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:18.732147+00:00 (in 9.999531 seconds)
notificationworker stdout | 2025-02-07 04:30:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:18 UTC)" (scheduled at 2025-02-07 04:30:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:30:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:30:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 8, 732896), True, datetime.datetime(2025, 2, 7, 4, 30, 8, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:30:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:30:08,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:30:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:30:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:30:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:24.423055+00:00 (in 13.998372 seconds)
gcworker stdout | 2025-02-07 04:30:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:40 UTC)" (scheduled at 2025-02-07 04:30:10.424250+00:00)
gcworker stdout | 2025-02-07 04:30:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:30:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902310431, None, 1, 0])
gcworker stdout | 2025-02-07 04:30:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:30:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:30:10,768 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:30:11,473 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:30:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:30:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:41.831521+00:00 (in 29.999483 seconds)
autopruneworker stdout | 2025-02-07 04:30:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:41 UTC)" (scheduled at 2025-02-07 04:30:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:30:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899011836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:30:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:30:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:30:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:41 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:30:11,869 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:30:13,220 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:30:14,140 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:30:14,207 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:30:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:30:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:29.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 04:30:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:29 UTC)" (scheduled at 2025-02-07 04:30:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:30:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:30:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:30:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:30:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:30:14,235 [235] [DEBUG] [app] Starting request: urn:request:94b161ba-7cb5-452c-bafe-b005f698f8fb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:14,235 [237] [DEBUG] [app] Starting request: urn:request:04103083-894a-43b6-a450-deb6c0dc0afd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:30:14,238 [239] [DEBUG] [app] Starting request: urn:request:90253fd8-16ee-4fdd-93a6-7666d866d0b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:30:14,238 [245] [DEBUG] [app] Starting request: urn:request:f99313fc-1419-4ea0-a04e-c3e494835e7f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:30:14,238 [239] [DEBUG] [app] Ending request: urn:request:90253fd8-16ee-4fdd-93a6-7666d866d0b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:90253fd8-16ee-4fdd-93a6-7666d866d0b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:30:14,238 [245] [DEBUG] [app] Ending request: urn:request:f99313fc-1419-4ea0-a04e-c3e494835e7f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f99313fc-1419-4ea0-a04e-c3e494835e7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:30:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:30:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:14,241 [237] [DEBUG] [app] Starting request: urn:request:971270fc-903d-4cba-8e32-1dd857e0a878 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:14,241 [238] [DEBUG] [app] Starting request: urn:request:ca4a23be-3f0e-4e18-9259-af18809afc51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:14,241 [237] [DEBUG] [app] Ending request: urn:request:971270fc-903d-4cba-8e32-1dd857e0a878 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:971270fc-903d-4cba-8e32-1dd857e0a878', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:30:14,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:14,241 [238] [DEBUG] [app] Ending request: urn:request:ca4a23be-3f0e-4e18-9259-af18809afc51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ca4a23be-3f0e-4e18-9259-af18809afc51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:30:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:30:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:14,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:14,252 [237] [DEBUG] [app] Ending request: urn:request:04103083-894a-43b6-a450-deb6c0dc0afd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04103083-894a-43b6-a450-deb6c0dc0afd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:30:14,252 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:14,252 [235] [DEBUG] [app] Ending request: urn:request:94b161ba-7cb5-452c-bafe-b005f698f8fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94b161ba-7cb5-452c-bafe-b005f698f8fb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:30:14,252 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:30:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:30:15,226 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:15,774 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:30:16,250 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:30:16,281 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:30:16,387 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:30:16,772 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:30:16,775 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:30:16,779 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:30:16,782 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:30:16,784 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:30:16,854 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:30:17,289 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:30:17,402 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:30:17,460 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:30:17,790 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:30:17,793 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:30:17,796 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:30:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:30:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:28.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 04:30:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:28 UTC)" (scheduled at 2025-02-07 04:30:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:30:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:30:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 18, 732790), True, datetime.datetime(2025, 2, 7, 4, 30, 18, 732790), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:30:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:30:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:30:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:30:18,979 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:30:18,989 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:30:18,992 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:30:18,995 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:30:18,997 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:30:19,002 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:30:19,005 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:30:19,008 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:30:19,010 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:30:19,134 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:30:19,199 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:30:19,561 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:30:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:30:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:03.419110+00:00 (in 39.997736 seconds)
repositorygcworker stdout | 2025-02-07 04:30:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:23 UTC)" (scheduled at 2025-02-07 04:30:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:30:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:30:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:30:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:30:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:40.424250+00:00 (in 16.000707 seconds)
gcworker stdout | 2025-02-07 04:30:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:54 UTC)" (scheduled at 2025-02-07 04:30:24.423055+00:00)
gcworker stdout | 2025-02-07 04:30:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:30:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:30:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:30:28,571 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:30:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:30:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:38.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 04:30:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:38 UTC)" (scheduled at 2025-02-07 04:30:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:30:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:30:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 28, 732818), True, datetime.datetime(2025, 2, 7, 4, 30, 28, 732818), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:30:28,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:30:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:30:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:30:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:30:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:44.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 04:30:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:44 UTC)" (scheduled at 2025-02-07 04:30:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:30:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:30:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:30:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:30:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:30:29,236 [238] [DEBUG] [app] Starting request: urn:request:c899b942-963e-4f0c-8353-4e5ac3d182de (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:29,236 [235] [DEBUG] [app] Starting request: urn:request:3185cf4f-c937-447d-a265-502318dd3423 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:30:29,238 [239] [DEBUG] [app] Starting request: urn:request:33e540fc-8a3a-4e7d-a6c4-219fd1596bd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:30:29,238 [246] [DEBUG] [app] Starting request: urn:request:391e3b2a-a800-4cc6-9788-ec96f1500eb7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:30:29,239 [239] [DEBUG] [app] Ending request: urn:request:33e540fc-8a3a-4e7d-a6c4-219fd1596bd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:33e540fc-8a3a-4e7d-a6c4-219fd1596bd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:30:29,239 [246] [DEBUG] [app] Ending request: urn:request:391e3b2a-a800-4cc6-9788-ec96f1500eb7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:391e3b2a-a800-4cc6-9788-ec96f1500eb7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:30:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:30:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:29,241 [235] [DEBUG] [app] Starting request: urn:request:7edb5415-2edf-436e-a7b9-f9dda3b16978 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:29,241 [237] [DEBUG] [app] Starting request: urn:request:9dbdb888-b39e-463e-8860-019e2b22c9dd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:29,242 [235] [DEBUG] [app] Ending request: urn:request:7edb5415-2edf-436e-a7b9-f9dda3b16978 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7edb5415-2edf-436e-a7b9-f9dda3b16978', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:30:29,242 [237] [DEBUG] [app] Ending request: urn:request:9dbdb888-b39e-463e-8860-019e2b22c9dd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9dbdb888-b39e-463e-8860-019e2b22c9dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:30:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:30:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:30:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:29,253 [235] [DEBUG] [app] Ending request: urn:request:3185cf4f-c937-447d-a265-502318dd3423 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3185cf4f-c937-447d-a265-502318dd3423', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:30:29,253 [238] [DEBUG] [app] Ending request: urn:request:c899b942-963e-4f0c-8353-4e5ac3d182de (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c899b942-963e-4f0c-8353-4e5ac3d182de', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:30:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:30:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:30:30,328 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:30:31,506 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:30:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:30:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:32.625945+00:00 (in 1.004356 seconds)
namespacegcworker stdout | 2025-02-07 04:30:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:31 UTC)" (scheduled at 2025-02-07 04:30:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:30:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:30:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 31, 621865), True, datetime.datetime(2025, 2, 7, 4, 30, 31, 621865), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:30:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:30:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:30:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:30:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:30:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:31.621131+00:00 (in 58.994705 seconds)
namespacegcworker stdout | 2025-02-07 04:30:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:32 UTC)" (scheduled at 2025-02-07 04:30:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:30:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:30:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:30:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:30:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:03.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 04:30:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:03 UTC)" (scheduled at 2025-02-07 04:30:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:30:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 30, 33, 919996), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:30:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:30:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:30:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:30:35,788 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:30:36,879 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:56.713547+00:00 (in 18.997203 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:37 UTC)" (scheduled at 2025-02-07 04:30:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:30:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:30:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:48.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:30:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:48 UTC)" (scheduled at 2025-02-07 04:30:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:30:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:30:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 38, 732813), True, datetime.datetime(2025, 2, 7, 4, 30, 38, 732813), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:30:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:30:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:30:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:30:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:30:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:54.423055+00:00 (in 13.998359 seconds)
gcworker stdout | 2025-02-07 04:30:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:10 UTC)" (scheduled at 2025-02-07 04:30:40.424250+00:00)
gcworker stdout | 2025-02-07 04:30:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:30:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902340431, None, 1, 0])
gcworker stdout | 2025-02-07 04:30:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:30:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:30:40,805 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:30:41,506 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:30:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:30:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:11.831521+00:00 (in 29.999521 seconds)
autopruneworker stdout | 2025-02-07 04:30:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:11 UTC)" (scheduled at 2025-02-07 04:30:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:30:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899041836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:30:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:30:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:30:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:11 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:30:41,887 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:30:43,256 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:30:44,154 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:30:44,217 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
quotatotalworker stdout | 2025-02-07 04:30:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:30:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:59.225378+00:00 (in 14.999593 seconds)
quotatotalworker stdout | 2025-02-07 04:30:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:59 UTC)" (scheduled at 2025-02-07 04:30:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:30:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:30:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:30:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:30:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:30:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:30:44,236 [237] [DEBUG] [app] Starting request: urn:request:9583aeb8-b865-4c62-8ebc-6c7e2dc2d80a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:44,236 [235] [DEBUG] [app] Starting request: urn:request:9a64a4e9-1a19-4059-8747-2dd921b631a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:30:44,238 [239] [DEBUG] [app] Starting request: urn:request:336557a1-01f5-40b6-9fc0-896d0e49cd98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:30:44,239 [246] [DEBUG] [app] Starting request: urn:request:a18cb5f3-ace2-4114-bd78-d4c060c96d97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:30:44,239 [239] [DEBUG] [app] Ending request: urn:request:336557a1-01f5-40b6-9fc0-896d0e49cd98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:336557a1-01f5-40b6-9fc0-896d0e49cd98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:30:44,239 [246] [DEBUG] [app] Ending request: urn:request:a18cb5f3-ace2-4114-bd78-d4c060c96d97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a18cb5f3-ace2-4114-bd78-d4c060c96d97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:30:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:30:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:44,241 [237] [DEBUG] [app] Starting request: urn:request:59cd8337-fe89-498c-87cc-e985cb30f5a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:44,242 [236] [DEBUG] [app] Starting request: urn:request:f1101fbe-e7d0-40c8-b5c1-f4d92c284434 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:44,242 [236] [DEBUG] [app] Ending request: urn:request:f1101fbe-e7d0-40c8-b5c1-f4d92c284434 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f1101fbe-e7d0-40c8-b5c1-f4d92c284434', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:30:44,242 [237] [DEBUG] [app] Ending request: urn:request:59cd8337-fe89-498c-87cc-e985cb30f5a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59cd8337-fe89-498c-87cc-e985cb30f5a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:30:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:44,253 [237] [DEBUG] [app] Ending request: urn:request:9583aeb8-b865-4c62-8ebc-6c7e2dc2d80a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9583aeb8-b865-4c62-8ebc-6c7e2dc2d80a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:30:44,253 [235] [DEBUG] [app] Ending request: urn:request:9a64a4e9-1a19-4059-8747-2dd921b631a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a64a4e9-1a19-4059-8747-2dd921b631a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:30:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:30:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
autopruneworker stdout | 2025-02-07 04:30:45,244 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:45,810 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:30:46,274 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:30:46,292 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:30:46,423 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:30:46,782 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:30:46,785 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:30:46,789 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:30:46,792 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:30:46,795 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:30:46,890 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:30:47,314 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:30:47,438 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:30:47,496 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:30:47,799 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:30:47,802 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:30:47,805 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:30:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:30:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:50.814677+00:00 (in 2.082067 seconds)
notificationworker stdout | 2025-02-07 04:30:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:58 UTC)" (scheduled at 2025-02-07 04:30:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:30:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:30:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 48, 732817), True, datetime.datetime(2025, 2, 7, 4, 30, 48, 732817), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:30:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:30:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:30:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:30:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:30:48,992 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:30:49,000 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:30:49,005 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:30:49,008 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:30:49,011 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:30:49,014 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:30:49,017 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:30:49,020 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:30:49,023 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:30:49,147 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:30:49,210 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:30:49,579 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:30:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:30:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:30:58.732147+00:00 (in 7.917000 seconds)
notificationworker stdout | 2025-02-07 04:30:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:50 UTC)" (scheduled at 2025-02-07 04:30:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:30:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:30:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:30:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:30:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:10.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 04:30:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:24 UTC)" (scheduled at 2025-02-07 04:30:54.423055+00:00)
gcworker stdout | 2025-02-07 04:30:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:30:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:30:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:30:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:56.118855+00:00 (in 59.999536 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:30:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:56 UTC)" (scheduled at 2025-02-07 04:30:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:30:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:30:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:30:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:37.715878+00:00 (in 41.001863 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:56 UTC)" (scheduled at 2025-02-07 04:30:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:30:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:30:58,607 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:30:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:30:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:08.732147+00:00 (in 9.999504 seconds)
notificationworker stdout | 2025-02-07 04:30:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:08 UTC)" (scheduled at 2025-02-07 04:30:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:30:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:30:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 30, 58, 732860), True, datetime.datetime(2025, 2, 7, 4, 30, 58, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:30:58,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:30:58,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:30:58,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:30:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:30:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:14.225378+00:00 (in 14.999558 seconds)
quotatotalworker stdout | 2025-02-07 04:30:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:14 UTC)" (scheduled at 2025-02-07 04:30:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:30:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:30:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:30:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:30:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:30:59,236 [238] [DEBUG] [app] Starting request: urn:request:f53bff66-7843-42c9-b546-5a814b57e0ba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:59,236 [236] [DEBUG] [app] Starting request: urn:request:1731121b-66ef-4568-9124-6f3e48d5f285 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:30:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:59,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:30:59,239 [246] [DEBUG] [app] Starting request: urn:request:63b34cd6-f859-4eed-a076-65d8a4c406cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:30:59,239 [246] [DEBUG] [app] Ending request: urn:request:63b34cd6-f859-4eed-a076-65d8a4c406cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:63b34cd6-f859-4eed-a076-65d8a4c406cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:30:59,239 [244] [DEBUG] [app] Starting request: urn:request:4728f1eb-3b5e-4d79-907d-52f98c90d3e0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:30:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:30:59,240 [244] [DEBUG] [app] Ending request: urn:request:4728f1eb-3b5e-4d79-907d-52f98c90d3e0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4728f1eb-3b5e-4d79-907d-52f98c90d3e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:30:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:30:59,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:59,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:30:59,242 [236] [DEBUG] [app] Starting request: urn:request:3523f0ce-7345-489f-8f6b-71c0093ea21a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:59,243 [236] [DEBUG] [app] Ending request: urn:request:3523f0ce-7345-489f-8f6b-71c0093ea21a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3523f0ce-7345-489f-8f6b-71c0093ea21a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:30:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:30:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:30:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:59,244 [235] [DEBUG] [app] Starting request: urn:request:5114d398-ea8f-43f6-9a84-3c8276b46e6c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:30:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:59,244 [235] [DEBUG] [app] Ending request: urn:request:5114d398-ea8f-43f6-9a84-3c8276b46e6c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5114d398-ea8f-43f6-9a84-3c8276b46e6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:30:59,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:30:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:30:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:30:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:30:59,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:30:59,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:30:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:30:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:30:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:30:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:30:59,254 [236] [DEBUG] [app] Ending request: urn:request:1731121b-66ef-4568-9124-6f3e48d5f285 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1731121b-66ef-4568-9124-6f3e48d5f285', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:30:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:30:59,255 [238] [DEBUG] [app] Ending request: urn:request:f53bff66-7843-42c9-b546-5a814b57e0ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f53bff66-7843-42c9-b546-5a814b57e0ba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:30:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:30:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:30:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:30:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:30:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:30:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:04.936688+00:00 (in 4.998306 seconds)
exportactionlogsworker stdout | 2025-02-07 04:30:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:59 UTC)" (scheduled at 2025-02-07 04:30:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:30:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:30:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:31:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:31:00,361 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:31:01,520 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:31:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:31:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:23.420886+00:00 (in 20.001312 seconds)
repositorygcworker stdout | 2025-02-07 04:31:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:03 UTC)" (scheduled at 2025-02-07 04:31:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:31:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:31:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 3, 419845), True, datetime.datetime(2025, 2, 7, 4, 31, 3, 419845), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:31:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:31:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:31:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:31:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:31:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:33.919286+00:00 (in 29.999563 seconds)
buildlogsarchiver stdout | 2025-02-07 04:31:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:33 UTC)" (scheduled at 2025-02-07 04:31:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:31:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 31, 3, 919978), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:31:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:31:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:31:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:31:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:31:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:59.937899+00:00 (in 55.000785 seconds)
exportactionlogsworker stdout | 2025-02-07 04:31:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:04 UTC)" (scheduled at 2025-02-07 04:31:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:31:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:31:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 4, 937404), True, datetime.datetime(2025, 2, 7, 4, 31, 4, 937404), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:31:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:31:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:31:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:04 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,057 [55] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,058 [55] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:05.057762+00:00 (in 3599.999449 seconds)
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,058 [55] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-02-07 05:31:05 UTC)" (scheduled at 2025-02-07 04:31:05.057762+00:00)
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,058 [55] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,058 [55] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ...
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,062 [55] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,062 [55] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,062 [55] [DEBUG] [__main__] Performing blob upload cleanup
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,063 [55] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 5, 4, 31, 5, 62406), 1, 0])
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,069 [55] [DEBUG] [__main__] No additional stale blob uploads found
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,069 [55] [DEBUG] [data.database] Disconnecting from database.
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7fda72c4b8b0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7fda72c615e0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fda72c4f430>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7fda72c61430>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fda72c4f5e0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.openshift-storage.svc.cluster.local:443', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.regions] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7fda72c61940>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function add_recursion_detection_header at 0x7fda72c610d0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7fda72c4bca0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': 'datastorage/registry/uploads', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360/', 'url': 'https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fda6c1ad490>, 'has_streaming_input': False, 'auth_type': 'v4', 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'params': {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'Prefix': 'datastorage/registry/uploads', 'EncodingType': 'url'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}}
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,070 [55] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fda6c1ad3a0>>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.ListObjectsV2: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fda6ddd5b80>>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7fda72c61310>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.hooks] Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7fda72c4f550>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.auth] Calculating signature using v4 auth.
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.auth] CanonicalRequest:
blobuploadcleanupworker stdout | GET
blobuploadcleanupworker stdout | /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | encoding-type=url&list-type=2&prefix=datastorage%2Fregistry%2Fuploads
blobuploadcleanupworker stdout | host:s3.openshift-storage.svc.cluster.local
blobuploadcleanupworker stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | x-amz-date:20250207T043105Z
blobuploadcleanupworker stdout | host;x-amz-content-sha256;x-amz-date
blobuploadcleanupworker stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.auth] StringToSign:
blobuploadcleanupworker stdout | AWS4-HMAC-SHA256
blobuploadcleanupworker stdout | 20250207T043105Z
blobuploadcleanupworker stdout | 20250207/us-east-1/s3/aws4_request
blobuploadcleanupworker stdout | 120a8ed016d15552364c5f48c90ccb70eb5789446b59fd11ea5f14ccead5bb5c
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.auth] Signature:
blobuploadcleanupworker stdout | 96bba895152c4d1f034814031007fd9ba319a4b33ca5ef01049f60190cb6ad83
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler <function add_retry_headers at 0x7fda72c4f3a0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,071 [55] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url, headers={'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20250207T043105Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=pjqBQGfl18IdF9RWc0aJ/20250207/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=96bba895152c4d1f034814031007fd9ba319a4b33ca5ef01049f60190cb6ad83', 'amz-sdk-invocation-id': b'110d6113-cb97-476f-9dfd-394bad3f4d26', 'amz-sdk-request': b'attempt=1'}>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,072 [55] [DEBUG] [botocore.httpsession] Certificate path: /app/lib/python3.9/site-packages/certifi/cacert.pem
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,072 [55] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: s3.openshift-storage.svc.cluster.local
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,110 [55] [DEBUG] [urllib3.connectionpool] https://s3.openshift-storage.svc.cluster.local:443 "GET /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url HTTP/1.1" 200 411
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,110 [55] [DEBUG] [botocore.parsers] Response headers: {'Server': 'NooBaa/5.17.3-427bfd2', 'x-amz-request-id': 'm6u9qzk4-9r94f0-rd7', 'x-amz-id-2': 'm6u9qzk4-9r94f0-rd7', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Methods': 'GET,POST,PUT,DELETE,OPTIONS', 'Access-Control-Allow-Headers': 'Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256,amz-sdk-invocation-id,amz-sdk-request', 'Access-Control-Expose-Headers': 'ETag,X-Amz-Version-Id', 'Content-Type': 'application/xml', 'Content-Length': '411', 'Date': 'Fri, 07 Feb 2025 04:31:05 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5'}
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,110 [55] [DEBUG] [botocore.parsers] Response body:
blobuploadcleanupworker stdout | b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360</Name><Prefix>datastorage%2Fregistry%2Fuploads</Prefix><Delimiter>undefined</Delimiter><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><EncodingType>url</EncodingType><StartAfter>undefined</StartAfter><KeyCount>0</KeyCount></ListBucketResult>'
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,110 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda5fb4eac0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,110 [55] [DEBUG] [botocore.retryhandler] No retry needed.
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,110 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,110 [55] [DEBUG] [botocore.hooks] Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7fda72c4b9d0>
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,111 [55] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,111 [55] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,112 [55] [DEBUG] [util.locking] Released lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 04:31:05,112 [55] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-02-07 05:31:05 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:31:05,824 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:31:06,901 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:31:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:31:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:18.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 04:31:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:18 UTC)" (scheduled at 2025-02-07 04:31:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:31:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:31:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 8, 732858), True, datetime.datetime(2025, 2, 7, 4, 31, 8, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:31:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:31:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:31:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:31:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:31:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:24.423055+00:00 (in 13.998312 seconds)
gcworker stdout | 2025-02-07 04:31:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:40 UTC)" (scheduled at 2025-02-07 04:31:10.424250+00:00)
gcworker stdout | 2025-02-07 04:31:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:31:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902370430, None, 1, 0])
gcworker stdout | 2025-02-07 04:31:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:31:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:31:10,828 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:31:11,542 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:31:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:31:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:41.831521+00:00 (in 29.999539 seconds)
autopruneworker stdout | 2025-02-07 04:31:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:41 UTC)" (scheduled at 2025-02-07 04:31:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:31:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899071836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:31:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:31:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:31:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:41 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:31:11,907 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:31:13,292 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:31:14,176 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
quotatotalworker stdout | 2025-02-07 04:31:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:31:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:29.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 04:31:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:29 UTC)" (scheduled at 2025-02-07 04:31:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:31:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:31:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:31:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:31:14,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:31:14,235 [235] [DEBUG] [app] Starting request: urn:request:676fbfd3-b74e-42e4-a633-384510a5fe20 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:14,235 [236] [DEBUG] [app] Starting request: urn:request:597f281e-c17d-4d53-9f58-eb3808220278 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:31:14,238 [244] [DEBUG] [app] Starting request: urn:request:b176ae99-3cc8-43a5-9dcd-2c72d8318b82 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:14,238 [239] [DEBUG] [app] Starting request: urn:request:cc84dea5-d43b-4f5a-8e3b-186076753883 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:14,239 [239] [DEBUG] [app] Ending request: urn:request:cc84dea5-d43b-4f5a-8e3b-186076753883 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc84dea5-d43b-4f5a-8e3b-186076753883', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:31:14,239 [244] [DEBUG] [app] Ending request: urn:request:b176ae99-3cc8-43a5-9dcd-2c72d8318b82 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b176ae99-3cc8-43a5-9dcd-2c72d8318b82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:31:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:31:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:31:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:14,242 [235] [DEBUG] [app] Starting request: urn:request:aef7ef30-0de4-4e14-89ca-2e5475f63e80 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:14,242 [238] [DEBUG] [app] Starting request: urn:request:7ede7790-d57e-4464-822b-a258c0bac72a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:14,242 [235] [DEBUG] [app] Ending request: urn:request:aef7ef30-0de4-4e14-89ca-2e5475f63e80 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aef7ef30-0de4-4e14-89ca-2e5475f63e80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:31:14,242 [238] [DEBUG] [app] Ending request: urn:request:7ede7790-d57e-4464-822b-a258c0bac72a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ede7790-d57e-4464-822b-a258c0bac72a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:31:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:31:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:31:14,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:14,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:31:14,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
chunkcleanupworker stdout | 2025-02-07 04:31:14,253 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 04:31:14,254 [235] [DEBUG] [app] Ending request: urn:request:676fbfd3-b74e-42e4-a633-384510a5fe20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:676fbfd3-b74e-42e4-a633-384510a5fe20', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:14,254 [236] [DEBUG] [app] Ending request: urn:request:597f281e-c17d-4d53-9f58-eb3808220278 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:597f281e-c17d-4d53-9f58-eb3808220278', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:31:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:31:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:31:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
autopruneworker stdout | 2025-02-07 04:31:15,280 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:15,839 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:31:16,300 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:31:16,310 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:31:16,449 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:31:16,791 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:31:16,793 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:31:16,796 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:31:16,799 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:31:16,802 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:31:16,918 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:31:17,350 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:31:17,468 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:31:17,523 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:31:17,808 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:31:17,812 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:31:17,814 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:31:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:31:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 04:31:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:28 UTC)" (scheduled at 2025-02-07 04:31:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:31:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:31:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 18, 732865), True, datetime.datetime(2025, 2, 7, 4, 31, 18, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:31:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:31:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:31:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:31:19,003 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:31:19,012 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:31:19,015 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:31:19,018 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:31:19,021 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:31:19,025 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:31:19,028 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:31:19,030 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:31:19,033 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:31:19,162 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:31:19,246 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:31:19,616 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:31:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:31:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:03.419110+00:00 (in 39.997752 seconds)
repositorygcworker stdout | 2025-02-07 04:31:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:23 UTC)" (scheduled at 2025-02-07 04:31:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:31:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:31:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:31:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:31:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:40.424250+00:00 (in 16.000722 seconds)
gcworker stdout | 2025-02-07 04:31:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:54 UTC)" (scheduled at 2025-02-07 04:31:24.423055+00:00)
gcworker stdout | 2025-02-07 04:31:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:31:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:31:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:31:28,640 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:31:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:31:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:38.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 04:31:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:38 UTC)" (scheduled at 2025-02-07 04:31:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:31:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:31:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 28, 732865), True, datetime.datetime(2025, 2, 7, 4, 31, 28, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:31:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:31:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:31:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:31:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:31:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:44.225378+00:00 (in 14.999534 seconds)
quotatotalworker stdout | 2025-02-07 04:31:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:44 UTC)" (scheduled at 2025-02-07 04:31:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:31:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:31:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:31:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:31:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:31:29,235 [235] [DEBUG] [app] Starting request: urn:request:bd2e9e5f-eabe-4563-81b4-bb28c43e0b85 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:29,235 [238] [DEBUG] [app] Starting request: urn:request:9fd4974f-0ae7-4ba1-a19c-288e051582ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:31:29,238 [239] [DEBUG] [app] Starting request: urn:request:c9644bc3-ac45-43c3-8d8a-a7f14b1ff448 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:29,238 [246] [DEBUG] [app] Starting request: urn:request:a8f07ceb-f9c6-4791-aebf-9373f207ae23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:29,238 [246] [DEBUG] [app] Ending request: urn:request:a8f07ceb-f9c6-4791-aebf-9373f207ae23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a8f07ceb-f9c6-4791-aebf-9373f207ae23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:31:29,238 [239] [DEBUG] [app] Ending request: urn:request:c9644bc3-ac45-43c3-8d8a-a7f14b1ff448 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c9644bc3-ac45-43c3-8d8a-a7f14b1ff448', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:31:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:31:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:31:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:29,241 [238] [DEBUG] [app] Starting request: urn:request:fdf0f211-2794-495d-8336-d1d55373e37c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:29,241 [238] [DEBUG] [app] Ending request: urn:request:fdf0f211-2794-495d-8336-d1d55373e37c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdf0f211-2794-495d-8336-d1d55373e37c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:31:29,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:31:29,241 [237] [DEBUG] [app] Starting request: urn:request:2120c2c4-abf4-4afc-9be6-770a5582c9f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:29,242 [237] [DEBUG] [app] Ending request: urn:request:2120c2c4-abf4-4afc-9be6-770a5582c9f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2120c2c4-abf4-4afc-9be6-770a5582c9f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:31:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:31:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:31:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:31:29,252 [238] [DEBUG] [app] Ending request: urn:request:9fd4974f-0ae7-4ba1-a19c-288e051582ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9fd4974f-0ae7-4ba1-a19c-288e051582ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:29,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:31:29,253 [235] [DEBUG] [app] Ending request: urn:request:bd2e9e5f-eabe-4563-81b4-bb28c43e0b85 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd2e9e5f-eabe-4563-81b4-bb28c43e0b85', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:31:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:31:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:31:30,397 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:31:31,555 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:31:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:31:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:32.625945+00:00 (in 1.004362 seconds)
namespacegcworker stdout | 2025-02-07 04:31:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:31 UTC)" (scheduled at 2025-02-07 04:31:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:31:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:31:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 31, 621849), True, datetime.datetime(2025, 2, 7, 4, 31, 31, 621849), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:31:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:31:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:31:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:31:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:31:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:13.624301+00:00 (in 40.997885 seconds)
namespacegcworker stdout | 2025-02-07 04:31:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:32 UTC)" (scheduled at 2025-02-07 04:31:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:31:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:31:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:31:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:31:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:03.919286+00:00 (in 29.999513 seconds)
buildlogsarchiver stdout | 2025-02-07 04:31:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:03 UTC)" (scheduled at 2025-02-07 04:31:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:31:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 31, 33, 920049), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:31:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:31:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:31:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:31:35,843 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:31:36,931 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:56.713547+00:00 (in 18.997200 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:37 UTC)" (scheduled at 2025-02-07 04:31:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:31:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:31:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:48.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 04:31:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:48 UTC)" (scheduled at 2025-02-07 04:31:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:31:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:31:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 38, 732868), True, datetime.datetime(2025, 2, 7, 4, 31, 38, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:31:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:31:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:31:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:31:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:31:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:54.423055+00:00 (in 13.998363 seconds)
gcworker stdout | 2025-02-07 04:31:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:10 UTC)" (scheduled at 2025-02-07 04:31:40.424250+00:00)
gcworker stdout | 2025-02-07 04:31:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:31:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902400431, None, 1, 0])
gcworker stdout | 2025-02-07 04:31:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:31:40,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:31:40,849 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:31:41,578 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:31:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:31:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:11.831521+00:00 (in 29.999510 seconds)
autopruneworker stdout | 2025-02-07 04:31:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:11 UTC)" (scheduled at 2025-02-07 04:31:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:31:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899101836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:31:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:31:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:31:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:11 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:31:41,930 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:31:43,328 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:31:44,213 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
quotatotalworker stdout | 2025-02-07 04:31:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:31:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:59.225378+00:00 (in 14.999503 seconds)
quotatotalworker stdout | 2025-02-07 04:31:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:59 UTC)" (scheduled at 2025-02-07 04:31:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:31:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:31:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:31:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:31:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:31:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:31:44,235 [235] [DEBUG] [app] Starting request: urn:request:349de02b-51d4-49b2-aa4b-afe8aac902b8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:44,235 [238] [DEBUG] [app] Starting request: urn:request:16f9da95-94c3-42c0-8994-1fabc2b77277 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:31:44,239 [244] [DEBUG] [app] Starting request: urn:request:323eedba-a330-467f-8e9f-cfc1a932889e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:44,239 [239] [DEBUG] [app] Starting request: urn:request:ec3e2cdb-177b-46a3-9ef5-c60fc9d719de (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:44,239 [239] [DEBUG] [app] Ending request: urn:request:ec3e2cdb-177b-46a3-9ef5-c60fc9d719de (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec3e2cdb-177b-46a3-9ef5-c60fc9d719de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:31:44,239 [244] [DEBUG] [app] Ending request: urn:request:323eedba-a330-467f-8e9f-cfc1a932889e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:323eedba-a330-467f-8e9f-cfc1a932889e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:31:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:31:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:31:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:44,242 [237] [DEBUG] [app] Starting request: urn:request:26bc4e40-15db-4bdb-a9a7-cef444072a1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:44,243 [237] [DEBUG] [app] Ending request: urn:request:26bc4e40-15db-4bdb-a9a7-cef444072a1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26bc4e40-15db-4bdb-a9a7-cef444072a1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:31:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:31:44,244 [238] [DEBUG] [app] Starting request: urn:request:98cc3f43-540e-4231-8494-f576bb106080 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:44,244 [238] [DEBUG] [app] Ending request: urn:request:98cc3f43-540e-4231-8494-f576bb106080 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98cc3f43-540e-4231-8494-f576bb106080', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:31:44,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:31:44,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:44,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:44,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:44,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:44,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:44,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:31:44,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:31:44,257 [238] [DEBUG] [app] Ending request: urn:request:16f9da95-94c3-42c0-8994-1fabc2b77277 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16f9da95-94c3-42c0-8994-1fabc2b77277', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:44,257 [235] [DEBUG] [app] Ending request: urn:request:349de02b-51d4-49b2-aa4b-afe8aac902b8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:349de02b-51d4-49b2-aa4b-afe8aac902b8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:44,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:31:44,258 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gunicorn-web stdout | 2025-02-07 04:31:44,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:31:44,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
chunkcleanupworker stdout | 2025-02-07 04:31:44,290 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:31:45,316 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:45,873 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:31:46,315 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:31:46,331 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:31:46,485 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:31:46,799 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:31:46,802 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:31:46,805 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:31:46,809 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:31:46,811 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:31:46,945 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:31:47,360 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:31:47,504 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:31:47,560 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:31:47,817 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:31:47,821 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:31:47,823 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:31:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:31:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:50.814677+00:00 (in 2.082061 seconds)
notificationworker stdout | 2025-02-07 04:31:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:58 UTC)" (scheduled at 2025-02-07 04:31:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:31:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:31:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 48, 732826), True, datetime.datetime(2025, 2, 7, 4, 31, 48, 732826), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:31:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:31:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:31:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:31:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:31:49,014 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:31:49,020 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:31:49,023 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:31:49,026 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:31:49,029 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:31:49,032 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:31:49,036 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:31:49,039 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:31:49,041 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:31:49,198 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:31:49,262 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:31:49,640 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:31:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:31:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:31:58.732147+00:00 (in 7.917006 seconds)
notificationworker stdout | 2025-02-07 04:31:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:50 UTC)" (scheduled at 2025-02-07 04:31:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:31:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:31:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:31:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:31:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:10.424250+00:00 (in 16.000687 seconds)
gcworker stdout | 2025-02-07 04:31:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:24 UTC)" (scheduled at 2025-02-07 04:31:54.423055+00:00)
gcworker stdout | 2025-02-07 04:31:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:31:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:31:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:31:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:56.118855+00:00 (in 59.999509 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:31:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:56 UTC)" (scheduled at 2025-02-07 04:31:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:31:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:31:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:31:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:37.715878+00:00 (in 41.001856 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:56 UTC)" (scheduled at 2025-02-07 04:31:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:31:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:56 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:31:58,677 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:31:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:31:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:08.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:31:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:08 UTC)" (scheduled at 2025-02-07 04:31:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:31:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:31:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 31, 58, 732874), True, datetime.datetime(2025, 2, 7, 4, 31, 58, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:31:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:31:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:31:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:31:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:31:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:14.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 04:31:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:14 UTC)" (scheduled at 2025-02-07 04:31:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:31:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:31:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:31:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:31:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:31:59,235 [238] [DEBUG] [app] Starting request: urn:request:dcb771da-0cc3-4edb-bbca-70ca49ef78c8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:59,235 [236] [DEBUG] [app] Starting request: urn:request:435a5183-3115-4458-b1a9-7cb15bd258b3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:31:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:31:59,238 [244] [DEBUG] [app] Starting request: urn:request:5a509cd3-dacc-4976-b844-4c0a36815161 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:59,238 [246] [DEBUG] [app] Starting request: urn:request:bb618a16-8e35-4e08-b617-1da4bfcba475 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:31:59,239 [244] [DEBUG] [app] Ending request: urn:request:5a509cd3-dacc-4976-b844-4c0a36815161 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a509cd3-dacc-4976-b844-4c0a36815161', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:31:59,239 [246] [DEBUG] [app] Ending request: urn:request:bb618a16-8e35-4e08-b617-1da4bfcba475 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bb618a16-8e35-4e08-b617-1da4bfcba475', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:31:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:31:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:31:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:31:59,241 [236] [DEBUG] [app] Starting request: urn:request:ab3ef279-2caf-49a3-8bd1-ef287cb02239 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:59,242 [236] [DEBUG] [app] Ending request: urn:request:ab3ef279-2caf-49a3-8bd1-ef287cb02239 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ab3ef279-2caf-49a3-8bd1-ef287cb02239', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:31:59,241 [237] [DEBUG] [app] Starting request: urn:request:28a5caff-8974-4906-9372-3044b6c95f67 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:31:59,242 [237] [DEBUG] [app] Ending request: urn:request:28a5caff-8974-4906-9372-3044b6c95f67 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28a5caff-8974-4906-9372-3044b6c95f67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:31:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:31:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:31:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:31:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:31:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:31:59,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:59,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:31:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:31:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:31:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:31:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:31:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:31:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:31:59,253 [238] [DEBUG] [app] Ending request: urn:request:dcb771da-0cc3-4edb-bbca-70ca49ef78c8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcb771da-0cc3-4edb-bbca-70ca49ef78c8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:59,253 [236] [DEBUG] [app] Ending request: urn:request:435a5183-3115-4458-b1a9-7cb15bd258b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:435a5183-3115-4458-b1a9-7cb15bd258b3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:31:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:31:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:31:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:31:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:31:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:31:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 04:31:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:31:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:04.936688+00:00 (in 4.998357 seconds)
exportactionlogsworker stdout | 2025-02-07 04:31:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:59 UTC)" (scheduled at 2025-02-07 04:31:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:31:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:31:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:32:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:32:00,407 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:32:01,591 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:32:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:32:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:23.420886+00:00 (in 20.001298 seconds)
repositorygcworker stdout | 2025-02-07 04:32:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:03 UTC)" (scheduled at 2025-02-07 04:32:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:32:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:32:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 3, 419898), True, datetime.datetime(2025, 2, 7, 4, 32, 3, 419898), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:32:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:32:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:32:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:32:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:32:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:33.919286+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2025-02-07 04:32:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:33 UTC)" (scheduled at 2025-02-07 04:32:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:32:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 32, 3, 919964), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:32:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:32:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:32:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:32:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:32:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:59.937899+00:00 (in 55.000797 seconds)
exportactionlogsworker stdout | 2025-02-07 04:32:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:04 UTC)" (scheduled at 2025-02-07 04:32:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:32:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:32:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 4, 937386), True, datetime.datetime(2025, 2, 7, 4, 32, 4, 937386), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:32:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:32:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:32:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:32:05,878 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:32:06,967 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:32:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:18.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 04:32:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:18 UTC)" (scheduled at 2025-02-07 04:32:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:32:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:32:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 8, 732851), True, datetime.datetime(2025, 2, 7, 4, 32, 8, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:32:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:32:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:32:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:32:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:32:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:24.423055+00:00 (in 13.998301 seconds)
gcworker stdout | 2025-02-07 04:32:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:40 UTC)" (scheduled at 2025-02-07 04:32:10.424250+00:00)
gcworker stdout | 2025-02-07 04:32:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:32:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902430431, None, 1, 0])
gcworker stdout | 2025-02-07 04:32:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:32:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:32:10,874 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:32:11,617 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:32:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:32:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:41.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 04:32:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:41 UTC)" (scheduled at 2025-02-07 04:32:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:32:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899131836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:32:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:32:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:32:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:41 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:32:11,949 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:32:13,353 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
namespacegcworker stdout | 2025-02-07 04:32:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:32:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:31.621131+00:00 (in 17.996387 seconds)
namespacegcworker stdout | 2025-02-07 04:32:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:37:13 UTC)" (scheduled at 2025-02-07 04:32:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:32:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 32, 13, 625020), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:32:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 32, 13, 625020), True, datetime.datetime(2025, 2, 7, 4, 32, 13, 625020), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:32:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 32, 13, 625020), True, datetime.datetime(2025, 2, 7, 4, 32, 13, 625020), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 32, 13, 625020), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:32:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:32:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:37:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:32:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:32:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:29.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 04:32:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:29 UTC)" (scheduled at 2025-02-07 04:32:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:32:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:32:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:32:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:32:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:32:14,235 [236] [DEBUG] [app] Starting request: urn:request:a8590f62-4290-4396-ab5a-73e8f68ec190 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:14,235 [235] [DEBUG] [app] Starting request: urn:request:e7f6b492-2afe-4512-aa35-25b0606afbe4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:32:14,238 [245] [DEBUG] [app] Starting request: urn:request:ca5b5361-fd49-4008-b602-fab0a585d1d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:14,238 [241] [DEBUG] [app] Starting request: urn:request:6e553de6-062c-45c3-8317-33f5ad23e242 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:14,238 [245] [DEBUG] [app] Ending request: urn:request:ca5b5361-fd49-4008-b602-fab0a585d1d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ca5b5361-fd49-4008-b602-fab0a585d1d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:32:14,238 [241] [DEBUG] [app] Ending request: urn:request:6e553de6-062c-45c3-8317-33f5ad23e242 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e553de6-062c-45c3-8317-33f5ad23e242', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:32:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:32:14,239 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:14,241 [235] [DEBUG] [app] Starting request: urn:request:ba745662-a3c5-44ae-9f09-976ceff66109 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:14,241 [238] [DEBUG] [app] Starting request: urn:request:7b17eeda-1f66-4b02-bafa-9a727039ad64 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:14,242 [235] [DEBUG] [app] Ending request: urn:request:ba745662-a3c5-44ae-9f09-976ceff66109 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba745662-a3c5-44ae-9f09-976ceff66109', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:14,242 [238] [DEBUG] [app] Ending request: urn:request:7b17eeda-1f66-4b02-bafa-9a727039ad64 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7b17eeda-1f66-4b02-bafa-9a727039ad64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:32:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:14,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:32:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:32:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:32:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
notificationworker stdout | 2025-02-07 04:32:14,250 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gunicorn-web stdout | 2025-02-07 04:32:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:14,254 [235] [DEBUG] [app] Ending request: urn:request:e7f6b492-2afe-4512-aa35-25b0606afbe4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e7f6b492-2afe-4512-aa35-25b0606afbe4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:32:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:32:14,255 [236] [DEBUG] [app] Ending request: urn:request:a8590f62-4290-4396-ab5a-73e8f68ec190 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8590f62-4290-4396-ab5a-73e8f68ec190', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:32:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
chunkcleanupworker stdout | 2025-02-07 04:32:14,327 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:32:15,331 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:15,901 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:32:16,343 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:32:16,367 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:32:16,521 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:32:16,807 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:32:16,810 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:32:16,814 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:32:16,817 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:32:16,819 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:32:16,954 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:32:17,397 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:32:17,543 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:32:17,598 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:32:17,827 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:32:17,830 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:32:17,833 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:32:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:28.732147+00:00 (in 9.999540 seconds)
notificationworker stdout | 2025-02-07 04:32:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:28 UTC)" (scheduled at 2025-02-07 04:32:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:32:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:32:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 18, 732903), True, datetime.datetime(2025, 2, 7, 4, 32, 18, 732903), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:32:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:32:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:32:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:32:19,026 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:32:19,030 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:32:19,033 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:32:19,036 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:32:19,039 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:32:19,042 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:32:19,046 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:32:19,051 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:32:19,054 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:32:19,218 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:32:19,274 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:32:19,676 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:32:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:32:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:03.419110+00:00 (in 39.997678 seconds)
repositorygcworker stdout | 2025-02-07 04:32:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:23 UTC)" (scheduled at 2025-02-07 04:32:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:32:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:32:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:32:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:32:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:40.424250+00:00 (in 16.000677 seconds)
gcworker stdout | 2025-02-07 04:32:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:54 UTC)" (scheduled at 2025-02-07 04:32:24.423055+00:00)
gcworker stdout | 2025-02-07 04:32:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:32:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:32:54 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:32:28,707 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 04:32:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:33.814166+00:00 (in 5.081595 seconds)
notificationworker stdout | 2025-02-07 04:32:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:38 UTC)" (scheduled at 2025-02-07 04:32:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:32:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:32:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 28, 732882), True, datetime.datetime(2025, 2, 7, 4, 32, 28, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:32:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:32:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:32:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:32:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:32:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:44.225378+00:00 (in 14.999580 seconds)
quotatotalworker stdout | 2025-02-07 04:32:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:44 UTC)" (scheduled at 2025-02-07 04:32:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:32:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:32:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:32:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:32:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:32:29,235 [235] [DEBUG] [app] Starting request: urn:request:86254eab-91d4-41bf-a950-63d253abbe2c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:29,235 [237] [DEBUG] [app] Starting request: urn:request:b0a9d5b7-1b32-49fe-b773-569d4a6f2485 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:32:29,238 [246] [DEBUG] [app] Starting request: urn:request:ccc73d54-2c09-424d-a4d1-b2a5b0a0f437 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:29,239 [246] [DEBUG] [app] Ending request: urn:request:ccc73d54-2c09-424d-a4d1-b2a5b0a0f437 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ccc73d54-2c09-424d-a4d1-b2a5b0a0f437', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:32:29,239 [239] [DEBUG] [app] Starting request: urn:request:44104f68-4c76-4315-8436-1e5f4a70a4e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:32:29,239 [239] [DEBUG] [app] Ending request: urn:request:44104f68-4c76-4315-8436-1e5f4a70a4e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44104f68-4c76-4315-8436-1e5f4a70a4e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:32:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:29,242 [236] [DEBUG] [app] Starting request: urn:request:bf997103-fd6b-43f1-a75a-c21a08e35795 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:29,242 [238] [DEBUG] [app] Starting request: urn:request:abeece0a-d44a-4a6b-8485-29f89b1c88f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:29,242 [236] [DEBUG] [app] Ending request: urn:request:bf997103-fd6b-43f1-a75a-c21a08e35795 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf997103-fd6b-43f1-a75a-c21a08e35795', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:29,242 [238] [DEBUG] [app] Ending request: urn:request:abeece0a-d44a-4a6b-8485-29f89b1c88f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:abeece0a-d44a-4a6b-8485-29f89b1c88f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:32:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:32:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:32:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:32:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:32:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:29,254 [237] [DEBUG] [app] Ending request: urn:request:b0a9d5b7-1b32-49fe-b773-569d4a6f2485 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b0a9d5b7-1b32-49fe-b773-569d4a6f2485', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:29,254 [235] [DEBUG] [app] Ending request: urn:request:86254eab-91d4-41bf-a950-63d253abbe2c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:86254eab-91d4-41bf-a950-63d253abbe2c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:32:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:32:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:32:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:32:30,444 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:32:31,610 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:32:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:32:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:32.625945+00:00 (in 1.004434 seconds)
namespacegcworker stdout | 2025-02-07 04:32:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:31 UTC)" (scheduled at 2025-02-07 04:32:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:32:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:32:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 31, 621787), True, datetime.datetime(2025, 2, 7, 4, 32, 31, 621787), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:32:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:32:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:32:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:32:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:32:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:31.621131+00:00 (in 58.994711 seconds)
namespacegcworker stdout | 2025-02-07 04:32:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:32 UTC)" (scheduled at 2025-02-07 04:32:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:32:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:32:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:32:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:38.732147+00:00 (in 4.917539 seconds)
notificationworker stdout | 2025-02-07 04:32:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:37:33 UTC)" (scheduled at 2025-02-07 04:32:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:32:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 32, 33, 814869), 'notification/%'])
notificationworker stdout | 2025-02-07 04:32:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 32, 33, 814869), True, datetime.datetime(2025, 2, 7, 4, 32, 33, 814869), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:32:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 32, 33, 814869), True, datetime.datetime(2025, 2, 7, 4, 32, 33, 814869), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 32, 33, 814869), 'notification/%'])
notificationworker stdout | 2025-02-07 04:32:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:32:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:37:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:32:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:32:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:03.919286+00:00 (in 29.999556 seconds)
buildlogsarchiver stdout | 2025-02-07 04:32:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:03 UTC)" (scheduled at 2025-02-07 04:32:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:32:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 32, 33, 919976), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:32:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:32:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:32:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:32:35,907 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:32:37,004 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:56.713547+00:00 (in 18.997178 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:37 UTC)" (scheduled at 2025-02-07 04:32:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:32:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:48.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 04:32:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:48 UTC)" (scheduled at 2025-02-07 04:32:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:32:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:32:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 38, 732901), True, datetime.datetime(2025, 2, 7, 4, 32, 38, 732901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:32:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:32:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:32:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:32:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:32:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:54.423055+00:00 (in 13.998293 seconds)
gcworker stdout | 2025-02-07 04:32:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:10 UTC)" (scheduled at 2025-02-07 04:32:40.424250+00:00)
gcworker stdout | 2025-02-07 04:32:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:32:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902460431, None, 1, 0])
gcworker stdout | 2025-02-07 04:32:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:32:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:32:40,908 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:32:41,648 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:32:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:32:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:11.831521+00:00 (in 29.999518 seconds)
autopruneworker stdout | 2025-02-07 04:32:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:11 UTC)" (scheduled at 2025-02-07 04:32:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:32:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899161836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:32:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:32:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:32:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:11 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:32:41,985 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:32:43,363 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:32:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:32:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:59.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:32:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:59 UTC)" (scheduled at 2025-02-07 04:32:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:32:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:32:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:32:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:32:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:32:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:32:44,235 [238] [DEBUG] [app] Starting request: urn:request:164e99ea-fa32-49c3-97ed-7d0b6d572f7f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:44,235 [237] [DEBUG] [app] Starting request: urn:request:0947f1ef-61e2-46d1-b4d0-1522fc6c87bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:32:44,238 [244] [DEBUG] [app] Starting request: urn:request:70c7d55b-0069-4ed2-9f8c-56d847ef9356 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:44,238 [246] [DEBUG] [app] Starting request: urn:request:c735f2ad-5610-4f41-b9a2-066c30b09bec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:44,238 [246] [DEBUG] [app] Ending request: urn:request:c735f2ad-5610-4f41-b9a2-066c30b09bec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c735f2ad-5610-4f41-b9a2-066c30b09bec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:32:44,238 [244] [DEBUG] [app] Ending request: urn:request:70c7d55b-0069-4ed2-9f8c-56d847ef9356 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:70c7d55b-0069-4ed2-9f8c-56d847ef9356', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:32:44,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:32:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:44,241 [236] [DEBUG] [app] Starting request: urn:request:4781b26a-c2e7-4e0b-b47d-52273da4a7ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:44,241 [237] [DEBUG] [app] Starting request: urn:request:753c39af-f2fc-4f51-80f7-dffd3e448724 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:44,241 [236] [DEBUG] [app] Ending request: urn:request:4781b26a-c2e7-4e0b-b47d-52273da4a7ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4781b26a-c2e7-4e0b-b47d-52273da4a7ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:44,241 [237] [DEBUG] [app] Ending request: urn:request:753c39af-f2fc-4f51-80f7-dffd3e448724 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:753c39af-f2fc-4f51-80f7-dffd3e448724', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:32:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:32:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:44,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:32:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:32:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:32:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:44,253 [237] [DEBUG] [app] Ending request: urn:request:0947f1ef-61e2-46d1-b4d0-1522fc6c87bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0947f1ef-61e2-46d1-b4d0-1522fc6c87bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:32:44,253 [238] [DEBUG] [app] Ending request: urn:request:164e99ea-fa32-49c3-97ed-7d0b6d572f7f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:164e99ea-fa32-49c3-97ed-7d0b6d572f7f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:32:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:32:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:32:44,287 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:32:44,362 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:32:45,352 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:45,914 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:32:46,379 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:32:46,403 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:32:46,558 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:32:46,816 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:32:46,819 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:32:46,822 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:32:46,825 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:32:46,828 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:32:46,978 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:32:47,431 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:32:47,579 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:32:47,634 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:32:47,835 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:32:47,839 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:32:47,842 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:32:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:50.814677+00:00 (in 2.082086 seconds)
notificationworker stdout | 2025-02-07 04:32:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:58 UTC)" (scheduled at 2025-02-07 04:32:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:32:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:32:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 48, 732874), True, datetime.datetime(2025, 2, 7, 4, 32, 48, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:32:48,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:32:48,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:32:48,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:32:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:32:49,035 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:32:49,038 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:32:49,043 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:32:49,045 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:32:49,048 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:32:49,052 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:32:49,057 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:32:49,062 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:32:49,065 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:32:49,250 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:32:49,310 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:32:49,711 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:32:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:32:58.732147+00:00 (in 7.917012 seconds)
notificationworker stdout | 2025-02-07 04:32:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:50 UTC)" (scheduled at 2025-02-07 04:32:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:32:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:32:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:32:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:32:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:10.424250+00:00 (in 16.000686 seconds)
gcworker stdout | 2025-02-07 04:32:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:24 UTC)" (scheduled at 2025-02-07 04:32:54.423055+00:00)
gcworker stdout | 2025-02-07 04:32:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:32:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:32:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:32:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:56.118855+00:00 (in 59.999522 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:32:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:56 UTC)" (scheduled at 2025-02-07 04:32:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:32:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:32:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:32:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:37.715878+00:00 (in 41.001814 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:56 UTC)" (scheduled at 2025-02-07 04:32:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:32:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:32:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:32:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:08.732147+00:00 (in 9.999547 seconds)
notificationworker stdout | 2025-02-07 04:32:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:08 UTC)" (scheduled at 2025-02-07 04:32:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:32:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:32:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 32, 58, 732883), True, datetime.datetime(2025, 2, 7, 4, 32, 58, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:32:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:32:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:32:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:32:58,744 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:32:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:32:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:14.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 04:32:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:14 UTC)" (scheduled at 2025-02-07 04:32:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:32:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:32:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:32:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:32:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:32:59,235 [235] [DEBUG] [app] Starting request: urn:request:ce51adde-43a6-4c2e-970d-337ed7f189aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:59,235 [238] [DEBUG] [app] Starting request: urn:request:344b5e7d-7e9a-4ac4-b172-e56b499fa4c2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:32:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:32:59,238 [239] [DEBUG] [app] Starting request: urn:request:3cd39659-3cc8-4e38-9de8-1d5e9b996028 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:59,238 [246] [DEBUG] [app] Starting request: urn:request:cfa2dd46-7241-4206-ac5e-6f18230d5cfa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:32:59,238 [239] [DEBUG] [app] Ending request: urn:request:3cd39659-3cc8-4e38-9de8-1d5e9b996028 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3cd39659-3cc8-4e38-9de8-1d5e9b996028', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:32:59,238 [246] [DEBUG] [app] Ending request: urn:request:cfa2dd46-7241-4206-ac5e-6f18230d5cfa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cfa2dd46-7241-4206-ac5e-6f18230d5cfa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:32:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:32:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:32:59,241 [238] [DEBUG] [app] Starting request: urn:request:8cbc4833-0e8a-4cab-95e6-1485aa64bffc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:59,242 [237] [DEBUG] [app] Starting request: urn:request:a84c132b-4195-4004-9562-aaca476a5fc9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:32:59,242 [238] [DEBUG] [app] Ending request: urn:request:8cbc4833-0e8a-4cab-95e6-1485aa64bffc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8cbc4833-0e8a-4cab-95e6-1485aa64bffc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:32:59,242 [237] [DEBUG] [app] Ending request: urn:request:a84c132b-4195-4004-9562-aaca476a5fc9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a84c132b-4195-4004-9562-aaca476a5fc9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:32:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:32:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:32:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:32:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:32:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:32:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:32:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:32:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:32:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:32:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:32:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:32:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:32:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:32:59,254 [238] [DEBUG] [app] Ending request: urn:request:344b5e7d-7e9a-4ac4-b172-e56b499fa4c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:344b5e7d-7e9a-4ac4-b172-e56b499fa4c2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:59,254 [235] [DEBUG] [app] Ending request: urn:request:ce51adde-43a6-4c2e-970d-337ed7f189aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce51adde-43a6-4c2e-970d-337ed7f189aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:32:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:32:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:32:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:32:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:32:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:32:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:32:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:32:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:04.936688+00:00 (in 4.998353 seconds)
exportactionlogsworker stdout | 2025-02-07 04:32:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:59 UTC)" (scheduled at 2025-02-07 04:32:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:32:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:32:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:33:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:33:00,480 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:33:01,625 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:33:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:33:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:23.420886+00:00 (in 20.001259 seconds)
repositorygcworker stdout | 2025-02-07 04:33:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:03 UTC)" (scheduled at 2025-02-07 04:33:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:33:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:33:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 3, 419935), True, datetime.datetime(2025, 2, 7, 4, 33, 3, 419935), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:33:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:33:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:33:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:33:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:33:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:33.919286+00:00 (in 29.999568 seconds)
buildlogsarchiver stdout | 2025-02-07 04:33:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:33 UTC)" (scheduled at 2025-02-07 04:33:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:33:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 33, 3, 919893), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:33:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:33:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:33:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:33:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:33:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:59.937899+00:00 (in 55.000795 seconds)
exportactionlogsworker stdout | 2025-02-07 04:33:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:04 UTC)" (scheduled at 2025-02-07 04:33:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:33:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:33:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 4, 937399), True, datetime.datetime(2025, 2, 7, 4, 33, 4, 937399), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:33:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:33:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:33:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:33:05,932 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:33:07,011 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:33:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:33:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:18.732147+00:00 (in 9.999448 seconds)
notificationworker stdout | 2025-02-07 04:33:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:18 UTC)" (scheduled at 2025-02-07 04:33:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:33:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:33:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 8, 733085), True, datetime.datetime(2025, 2, 7, 4, 33, 8, 733085), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:33:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:33:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:33:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:33:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:33:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:24.423055+00:00 (in 13.998335 seconds)
gcworker stdout | 2025-02-07 04:33:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:40 UTC)" (scheduled at 2025-02-07 04:33:10.424250+00:00)
gcworker stdout | 2025-02-07 04:33:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:33:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902490431, None, 1, 0])
gcworker stdout | 2025-02-07 04:33:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:33:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:33:10,930 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:33:11,685 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:33:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:33:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:41.831521+00:00 (in 29.999522 seconds)
autopruneworker stdout | 2025-02-07 04:33:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:41 UTC)" (scheduled at 2025-02-07 04:33:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:33:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899191836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:33:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:33:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:33:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:41 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:33:12,014 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:33:13,399 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:33:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:33:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:29.225378+00:00 (in 14.999572 seconds)
quotatotalworker stdout | 2025-02-07 04:33:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:29 UTC)" (scheduled at 2025-02-07 04:33:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:33:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:33:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:33:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:33:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:33:14,236 [238] [DEBUG] [app] Starting request: urn:request:d478c1a2-0f69-4933-ae12-0e3c3285b9fb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:14,236 [236] [DEBUG] [app] Starting request: urn:request:10c80a88-9700-49ae-905b-cb19a08484b0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:14,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:33:14,239 [246] [DEBUG] [app] Starting request: urn:request:a732c421-dc9c-4639-909f-1f58e0d8bf3a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:33:14,239 [246] [DEBUG] [app] Ending request: urn:request:a732c421-dc9c-4639-909f-1f58e0d8bf3a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a732c421-dc9c-4639-909f-1f58e0d8bf3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:33:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:33:14,240 [239] [DEBUG] [app] Starting request: urn:request:5690aa48-bf9f-444a-8213-e2a457062645 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:33:14,240 [239] [DEBUG] [app] Ending request: urn:request:5690aa48-bf9f-444a-8213-e2a457062645 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5690aa48-bf9f-444a-8213-e2a457062645', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:33:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:14,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:14,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:14,243 [237] [DEBUG] [app] Starting request: urn:request:bc81c470-eb62-4959-9bc2-b1a64bb0c40c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:14,243 [235] [DEBUG] [app] Starting request: urn:request:6cd3a87a-598b-4abc-992d-06cae8bbec43 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:14,243 [237] [DEBUG] [app] Ending request: urn:request:bc81c470-eb62-4959-9bc2-b1a64bb0c40c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc81c470-eb62-4959-9bc2-b1a64bb0c40c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:33:14,244 [235] [DEBUG] [app] Ending request: urn:request:6cd3a87a-598b-4abc-992d-06cae8bbec43 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6cd3a87a-598b-4abc-992d-06cae8bbec43', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:33:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:14,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:14,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:33:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:33:14,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:14,255 [236] [DEBUG] [app] Ending request: urn:request:10c80a88-9700-49ae-905b-cb19a08484b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10c80a88-9700-49ae-905b-cb19a08484b0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:14,255 [238] [DEBUG] [app] Ending request: urn:request:d478c1a2-0f69-4933-ae12-0e3c3285b9fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d478c1a2-0f69-4933-ae12-0e3c3285b9fb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:33:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:33:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:33:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:33:14,323 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:33:14,373 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:33:15,387 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:15,951 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:33:16,405 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:33:16,419 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:33:16,578 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:33:16,823 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:33:16,826 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:33:16,830 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:33:16,833 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:33:16,836 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:33:17,012 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:33:17,467 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:33:17,615 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:33:17,670 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:33:17,844 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:33:17,847 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:33:17,849 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:33:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:33:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:28.732147+00:00 (in 9.999517 seconds)
notificationworker stdout | 2025-02-07 04:33:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:28 UTC)" (scheduled at 2025-02-07 04:33:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:33:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:33:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 18, 732914), True, datetime.datetime(2025, 2, 7, 4, 33, 18, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:33:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:33:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:33:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:33:19,045 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:33:19,049 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:33:19,054 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:33:19,057 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:33:19,060 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:33:19,063 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:33:19,066 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:33:19,073 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:33:19,075 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:33:19,286 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:33:19,316 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:33:19,747 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:33:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:33:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:32.420707+00:00 (in 8.999313 seconds)
repositorygcworker stdout | 2025-02-07 04:33:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:23 UTC)" (scheduled at 2025-02-07 04:33:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:33:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:33:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:33:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:33:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:40.424250+00:00 (in 16.000766 seconds)
gcworker stdout | 2025-02-07 04:33:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:54 UTC)" (scheduled at 2025-02-07 04:33:24.423055+00:00)
gcworker stdout | 2025-02-07 04:33:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:33:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:33:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:33:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:33:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:38.732147+00:00 (in 9.999482 seconds)
notificationworker stdout | 2025-02-07 04:33:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:38 UTC)" (scheduled at 2025-02-07 04:33:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:33:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:33:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 28, 733009), True, datetime.datetime(2025, 2, 7, 4, 33, 28, 733009), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:33:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:33:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:33:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:33:28,752 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:33:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:33:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:44.225378+00:00 (in 14.999577 seconds)
quotatotalworker stdout | 2025-02-07 04:33:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:44 UTC)" (scheduled at 2025-02-07 04:33:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:33:29,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:33:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:33:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:33:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:33:29,235 [238] [DEBUG] [app] Starting request: urn:request:246e2083-bc54-459d-8f9d-c12f070786ac (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:29,235 [235] [DEBUG] [app] Starting request: urn:request:41c79cbb-174c-4282-bdf4-c24ee921665a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:33:29,238 [239] [DEBUG] [app] Starting request: urn:request:59a1abd6-522c-48c5-9ee5-88c0b618cdae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:33:29,238 [245] [DEBUG] [app] Starting request: urn:request:94c4e32f-75fa-4894-bdf0-e4016ab870e7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:33:29,238 [245] [DEBUG] [app] Ending request: urn:request:94c4e32f-75fa-4894-bdf0-e4016ab870e7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:94c4e32f-75fa-4894-bdf0-e4016ab870e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:33:29,238 [239] [DEBUG] [app] Ending request: urn:request:59a1abd6-522c-48c5-9ee5-88c0b618cdae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59a1abd6-522c-48c5-9ee5-88c0b618cdae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:33:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:33:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:33:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:29,241 [238] [DEBUG] [app] Starting request: urn:request:2dc3f726-546e-4a39-8b3f-4e0192aeb8a8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:29,242 [237] [DEBUG] [app] Starting request: urn:request:21f0200d-9a5f-4f4e-8ebb-6719ca57adf8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:29,242 [238] [DEBUG] [app] Ending request: urn:request:2dc3f726-546e-4a39-8b3f-4e0192aeb8a8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2dc3f726-546e-4a39-8b3f-4e0192aeb8a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:33:29,242 [237] [DEBUG] [app] Ending request: urn:request:21f0200d-9a5f-4f4e-8ebb-6719ca57adf8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21f0200d-9a5f-4f4e-8ebb-6719ca57adf8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:33:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:33:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:33:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:33:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:29,254 [235] [DEBUG] [app] Ending request: urn:request:41c79cbb-174c-4282-bdf4-c24ee921665a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:41c79cbb-174c-4282-bdf4-c24ee921665a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:29,254 [238] [DEBUG] [app] Ending request: urn:request:246e2083-bc54-459d-8f9d-c12f070786ac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:246e2083-bc54-459d-8f9d-c12f070786ac', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:33:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:33:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:33:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:33:30,511 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:33:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:33:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:32.625945+00:00 (in 1.004365 seconds)
namespacegcworker stdout | 2025-02-07 04:33:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:31 UTC)" (scheduled at 2025-02-07 04:33:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:33:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:33:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 31, 621851), True, datetime.datetime(2025, 2, 7, 4, 33, 31, 621851), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:33:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:33:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:33:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:33:31,657 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:33:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:33:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:03.419110+00:00 (in 30.997961 seconds)
repositorygcworker stdout | 2025-02-07 04:33:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:38:32 UTC)" (scheduled at 2025-02-07 04:33:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:33:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 33, 32, 421417), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:33:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 33, 32, 421417), True, datetime.datetime(2025, 2, 7, 4, 33, 32, 421417), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:33:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 33, 32, 421417), True, datetime.datetime(2025, 2, 7, 4, 33, 32, 421417), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 33, 32, 421417), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:33:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:33:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:38:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:33:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:33:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:31.621131+00:00 (in 58.994741 seconds)
namespacegcworker stdout | 2025-02-07 04:33:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:32 UTC)" (scheduled at 2025-02-07 04:33:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:33:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:33:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:33:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:33:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:03.919286+00:00 (in 29.999578 seconds)
buildlogsarchiver stdout | 2025-02-07 04:33:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:03 UTC)" (scheduled at 2025-02-07 04:33:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:33:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 33, 33, 919955), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:33:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:33:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:33:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:33:35,968 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:33:37,044 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:56.713547+00:00 (in 18.997159 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:37 UTC)" (scheduled at 2025-02-07 04:33:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:33:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:33:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:48.732147+00:00 (in 9.999516 seconds)
notificationworker stdout | 2025-02-07 04:33:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:48 UTC)" (scheduled at 2025-02-07 04:33:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:33:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:33:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 38, 732932), True, datetime.datetime(2025, 2, 7, 4, 33, 38, 732932), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:33:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:33:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:33:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:33:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:33:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:54.423055+00:00 (in 13.998344 seconds)
gcworker stdout | 2025-02-07 04:33:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:10 UTC)" (scheduled at 2025-02-07 04:33:40.424250+00:00)
gcworker stdout | 2025-02-07 04:33:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:33:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902520431, None, 1, 0])
gcworker stdout | 2025-02-07 04:33:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:33:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:33:40,968 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:33:41,701 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:33:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:33:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:11.831521+00:00 (in 29.999552 seconds)
autopruneworker stdout | 2025-02-07 04:33:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:11 UTC)" (scheduled at 2025-02-07 04:33:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:33:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899221836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:33:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:33:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:33:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:11 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:33:42,050 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:33:43,435 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:33:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:33:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:59.225378+00:00 (in 14.999429 seconds)
quotatotalworker stdout | 2025-02-07 04:33:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:59 UTC)" (scheduled at 2025-02-07 04:33:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:33:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:33:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:33:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:33:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:33:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:33:44,236 [237] [DEBUG] [app] Starting request: urn:request:9ee0886e-c083-43c8-b7ae-d152f4ee2fa3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:44,236 [238] [DEBUG] [app] Starting request: urn:request:0c21934b-08bb-459e-bca8-33522d8a34ae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:44,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:44,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:33:44,240 [244] [DEBUG] [app] Starting request: urn:request:45c0f023-df55-4f8c-a70d-b169a9d7fccc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:33:44,240 [239] [DEBUG] [app] Starting request: urn:request:fc2e9a15-7ff0-45c0-a823-01fa414c353a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:33:44,241 [239] [DEBUG] [app] Ending request: urn:request:fc2e9a15-7ff0-45c0-a823-01fa414c353a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc2e9a15-7ff0-45c0-a823-01fa414c353a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:33:44,241 [244] [DEBUG] [app] Ending request: urn:request:45c0f023-df55-4f8c-a70d-b169a9d7fccc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:45c0f023-df55-4f8c-a70d-b169a9d7fccc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:33:44,241 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:33:44,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:44,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:33:44,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:44,244 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:44,244 [238] [DEBUG] [app] Starting request: urn:request:56673c9b-a4dc-4131-8be0-356481275111 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:44,245 [238] [DEBUG] [app] Ending request: urn:request:56673c9b-a4dc-4131-8be0-356481275111 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56673c9b-a4dc-4131-8be0-356481275111', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:33:44,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:44,245 [235] [DEBUG] [app] Starting request: urn:request:360fc3b1-5328-46f8-8dec-0b5e9560edce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:44,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:44,246 [235] [DEBUG] [app] Ending request: urn:request:360fc3b1-5328-46f8-8dec-0b5e9560edce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:360fc3b1-5328-46f8-8dec-0b5e9560edce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:33:44,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:33:44,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:44,247 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:44,247 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:44,247 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:33:44,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:44,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:44,251 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:44,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:44,255 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:44,256 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:44,257 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:44,257 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:44,258 [238] [DEBUG] [app] Ending request: urn:request:0c21934b-08bb-459e-bca8-33522d8a34ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c21934b-08bb-459e-bca8-33522d8a34ae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:44,258 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
gunicorn-web stdout | 2025-02-07 04:33:44,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:33:44,259 [237] [DEBUG] [app] Ending request: urn:request:9ee0886e-c083-43c8-b7ae-d152f4ee2fa3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ee0886e-c083-43c8-b7ae-d152f4ee2fa3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:44,259 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:33:44,260 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.025)
notificationworker stdout | 2025-02-07 04:33:44,345 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:33:44,409 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:33:45,419 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:45,980 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:33:46,436 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:33:46,438 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:33:46,588 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:33:46,831 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:33:46,836 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:33:46,838 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:33:46,841 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:33:46,843 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:33:47,048 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:33:47,503 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:33:47,651 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:33:47,706 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:33:47,852 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:33:47,855 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:33:47,858 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:33:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:33:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:50.814677+00:00 (in 2.082062 seconds)
notificationworker stdout | 2025-02-07 04:33:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:58 UTC)" (scheduled at 2025-02-07 04:33:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:33:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:33:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 48, 732898), True, datetime.datetime(2025, 2, 7, 4, 33, 48, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:33:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:33:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:33:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:33:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:33:49,053 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:33:49,063 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:33:49,067 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:33:49,070 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:33:49,073 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:33:49,076 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:33:49,080 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:33:49,084 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:33:49,086 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:33:49,322 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:33:49,343 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:33:49,758 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:33:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:33:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:33:58.732147+00:00 (in 7.917009 seconds)
notificationworker stdout | 2025-02-07 04:33:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:50 UTC)" (scheduled at 2025-02-07 04:33:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:33:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:33:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:33:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:33:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:10.424250+00:00 (in 16.000678 seconds)
gcworker stdout | 2025-02-07 04:33:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:24 UTC)" (scheduled at 2025-02-07 04:33:54.423055+00:00)
gcworker stdout | 2025-02-07 04:33:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:33:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:33:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:33:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:56.118855+00:00 (in 59.999475 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:33:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:56 UTC)" (scheduled at 2025-02-07 04:33:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:33:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:33:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:33:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:37.715878+00:00 (in 41.001860 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:56 UTC)" (scheduled at 2025-02-07 04:33:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:33:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:33:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:33:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:08.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 04:33:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:08 UTC)" (scheduled at 2025-02-07 04:33:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:33:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:33:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 33, 58, 732891), True, datetime.datetime(2025, 2, 7, 4, 33, 58, 732891), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:33:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:33:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:33:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:33:58,770 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:33:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:33:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:14.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 04:33:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:14 UTC)" (scheduled at 2025-02-07 04:33:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:33:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:33:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:33:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:33:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:33:59,235 [237] [DEBUG] [app] Starting request: urn:request:e0da089e-6923-4d83-8bc1-e9d060473f57 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:59,235 [235] [DEBUG] [app] Starting request: urn:request:5614eb7a-3ec5-4a3c-a80a-68540cb16903 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:33:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:33:59,238 [239] [DEBUG] [app] Starting request: urn:request:a500b056-a85f-48ae-b30c-abad41a68222 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:33:59,238 [245] [DEBUG] [app] Starting request: urn:request:04ab5a47-4c4f-4951-b661-16f65ac2d7d1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:33:59,238 [245] [DEBUG] [app] Ending request: urn:request:04ab5a47-4c4f-4951-b661-16f65ac2d7d1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04ab5a47-4c4f-4951-b661-16f65ac2d7d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:33:59,238 [239] [DEBUG] [app] Ending request: urn:request:a500b056-a85f-48ae-b30c-abad41a68222 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a500b056-a85f-48ae-b30c-abad41a68222', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:33:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:33:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:33:59,241 [236] [DEBUG] [app] Starting request: urn:request:e24b371d-2159-46b4-854d-c44ddee06331 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:59,241 [237] [DEBUG] [app] Starting request: urn:request:36e3c91f-0135-4d35-9814-747c50ceb843 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:33:59,241 [236] [DEBUG] [app] Ending request: urn:request:e24b371d-2159-46b4-854d-c44ddee06331 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e24b371d-2159-46b4-854d-c44ddee06331', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:33:59,241 [237] [DEBUG] [app] Ending request: urn:request:36e3c91f-0135-4d35-9814-747c50ceb843 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36e3c91f-0135-4d35-9814-747c50ceb843', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:33:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:33:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:33:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:33:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:33:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:33:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:33:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:33:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:33:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:33:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:33:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:33:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:33:59,253 [235] [DEBUG] [app] Ending request: urn:request:5614eb7a-3ec5-4a3c-a80a-68540cb16903 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5614eb7a-3ec5-4a3c-a80a-68540cb16903', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:59,253 [237] [DEBUG] [app] Ending request: urn:request:e0da089e-6923-4d83-8bc1-e9d060473f57 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0da089e-6923-4d83-8bc1-e9d060473f57', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:33:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:33:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:33:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:33:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:33:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:33:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:33:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:33:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:04.936688+00:00 (in 4.998348 seconds)
exportactionlogsworker stdout | 2025-02-07 04:33:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:59 UTC)" (scheduled at 2025-02-07 04:33:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:33:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:33:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:34:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:34:00,523 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:34:01,669 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:34:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:34:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:23.420886+00:00 (in 20.001323 seconds)
repositorygcworker stdout | 2025-02-07 04:34:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:03 UTC)" (scheduled at 2025-02-07 04:34:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:34:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:34:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 3, 419867), True, datetime.datetime(2025, 2, 7, 4, 34, 3, 419867), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:34:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:34:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:34:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:34:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:34:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:33.919286+00:00 (in 29.999548 seconds)
buildlogsarchiver stdout | 2025-02-07 04:34:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:33 UTC)" (scheduled at 2025-02-07 04:34:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:34:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 34, 3, 919922), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:34:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:34:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:34:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:34:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:34:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:52.937648+00:00 (in 48.000531 seconds)
exportactionlogsworker stdout | 2025-02-07 04:34:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:04 UTC)" (scheduled at 2025-02-07 04:34:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:34:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:34:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 4, 937412), True, datetime.datetime(2025, 2, 7, 4, 34, 4, 937412), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:34:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:34:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:34:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:34:05,999 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:34:07,078 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:34:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:34:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:18.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 04:34:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:18 UTC)" (scheduled at 2025-02-07 04:34:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:34:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:34:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 8, 732873), True, datetime.datetime(2025, 2, 7, 4, 34, 8, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:34:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:34:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:34:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:34:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:34:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:24.423055+00:00 (in 13.998348 seconds)
gcworker stdout | 2025-02-07 04:34:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:40 UTC)" (scheduled at 2025-02-07 04:34:10.424250+00:00)
gcworker stdout | 2025-02-07 04:34:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:34:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902550431, None, 1, 0])
gcworker stdout | 2025-02-07 04:34:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:34:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:34:10,992 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:34:11,730 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:34:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:34:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 04:34:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:41 UTC)" (scheduled at 2025-02-07 04:34:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:34:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899251836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:34:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:34:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:34:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:41 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:34:12,087 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:34:13,445 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:34:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:34:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:29.225378+00:00 (in 14.999555 seconds)
quotatotalworker stdout | 2025-02-07 04:34:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:29 UTC)" (scheduled at 2025-02-07 04:34:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:34:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:34:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:34:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:34:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:34:14,235 [238] [DEBUG] [app] Starting request: urn:request:cee7da1d-ab27-4786-9e8a-a83a7fdebbb5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:14,235 [235] [DEBUG] [app] Starting request: urn:request:1f09a211-1568-408c-84fa-637f0ab6a5cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:34:14,238 [239] [DEBUG] [app] Starting request: urn:request:d6bd492b-6700-4ea0-bdc0-bf420331b34c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:14,238 [246] [DEBUG] [app] Starting request: urn:request:0282d55d-03dd-4e9d-9d10-7929deaf35c7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:14,239 [239] [DEBUG] [app] Ending request: urn:request:d6bd492b-6700-4ea0-bdc0-bf420331b34c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6bd492b-6700-4ea0-bdc0-bf420331b34c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:14,239 [246] [DEBUG] [app] Ending request: urn:request:0282d55d-03dd-4e9d-9d10-7929deaf35c7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0282d55d-03dd-4e9d-9d10-7929deaf35c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:34:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:34:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:14,241 [235] [DEBUG] [app] Starting request: urn:request:4a1f14c8-c2ed-474b-bde4-331199f55d38 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:14,242 [237] [DEBUG] [app] Starting request: urn:request:17be3757-dfb1-4771-9b2f-1cdb0126d21c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:14,242 [235] [DEBUG] [app] Ending request: urn:request:4a1f14c8-c2ed-474b-bde4-331199f55d38 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a1f14c8-c2ed-474b-bde4-331199f55d38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:34:14,242 [237] [DEBUG] [app] Ending request: urn:request:17be3757-dfb1-4771-9b2f-1cdb0126d21c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17be3757-dfb1-4771-9b2f-1cdb0126d21c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:34:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:34:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:34:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:14,253 [235] [DEBUG] [app] Ending request: urn:request:1f09a211-1568-408c-84fa-637f0ab6a5cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1f09a211-1568-408c-84fa-637f0ab6a5cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:14,253 [238] [DEBUG] [app] Ending request: urn:request:cee7da1d-ab27-4786-9e8a-a83a7fdebbb5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cee7da1d-ab27-4786-9e8a-a83a7fdebbb5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:34:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:34:14,353 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:34:14,445 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:34:15,443 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:16,017 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:34:16,452 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:34:16,469 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:34:16,625 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:34:16,840 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:34:16,843 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:34:16,847 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:34:16,851 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:34:16,853 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:34:17,085 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:34:17,532 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:34:17,687 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:34:17,718 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:34:17,861 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:34:17,865 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:34:17,868 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:34:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:34:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:28.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:34:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:28 UTC)" (scheduled at 2025-02-07 04:34:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:34:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:34:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 18, 732874), True, datetime.datetime(2025, 2, 7, 4, 34, 18, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:34:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:34:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:34:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:34:19,060 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:34:19,072 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:34:19,076 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:34:19,079 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:34:19,083 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:34:19,086 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:34:19,090 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:34:19,093 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:34:19,095 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:34:19,341 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:34:19,379 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:34:19,794 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:34:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:34:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:03.419110+00:00 (in 39.997776 seconds)
repositorygcworker stdout | 2025-02-07 04:34:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:23 UTC)" (scheduled at 2025-02-07 04:34:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:34:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:34:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:34:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:34:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:40.424250+00:00 (in 16.000758 seconds)
gcworker stdout | 2025-02-07 04:34:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:54 UTC)" (scheduled at 2025-02-07 04:34:24.423055+00:00)
gcworker stdout | 2025-02-07 04:34:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:34:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:34:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:34:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:34:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:38.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 04:34:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:38 UTC)" (scheduled at 2025-02-07 04:34:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:34:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:34:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 28, 732858), True, datetime.datetime(2025, 2, 7, 4, 34, 28, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:34:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:34:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:34:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:34:28,806 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:34:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:34:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:44.225378+00:00 (in 14.999607 seconds)
quotatotalworker stdout | 2025-02-07 04:34:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:44 UTC)" (scheduled at 2025-02-07 04:34:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:34:29,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:34:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:34:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:34:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:34:29,235 [235] [DEBUG] [app] Starting request: urn:request:4b3e330d-61a8-4bfe-95c1-14b39269680f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:29,235 [238] [DEBUG] [app] Starting request: urn:request:f00f7372-2dfb-43bd-9595-b2c481b6c322 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:34:29,238 [239] [DEBUG] [app] Starting request: urn:request:69af2724-d182-468b-bb07-ede3e7f69705 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:29,238 [246] [DEBUG] [app] Starting request: urn:request:4b235bbe-696d-4985-afcc-bec84ea8a2b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:29,239 [239] [DEBUG] [app] Ending request: urn:request:69af2724-d182-468b-bb07-ede3e7f69705 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:69af2724-d182-468b-bb07-ede3e7f69705', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:29,239 [246] [DEBUG] [app] Ending request: urn:request:4b235bbe-696d-4985-afcc-bec84ea8a2b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4b235bbe-696d-4985-afcc-bec84ea8a2b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:34:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:34:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:29,242 [235] [DEBUG] [app] Starting request: urn:request:10fab9c0-e6ab-4d32-afc0-56e2c720a2bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:29,242 [236] [DEBUG] [app] Starting request: urn:request:d0334fce-f517-44c5-b21a-5d7714f729a9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:29,242 [235] [DEBUG] [app] Ending request: urn:request:10fab9c0-e6ab-4d32-afc0-56e2c720a2bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10fab9c0-e6ab-4d32-afc0-56e2c720a2bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:34:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:34:29,242 [236] [DEBUG] [app] Ending request: urn:request:d0334fce-f517-44c5-b21a-5d7714f729a9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d0334fce-f517-44c5-b21a-5d7714f729a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:34:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:34:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:34:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:29,254 [238] [DEBUG] [app] Ending request: urn:request:f00f7372-2dfb-43bd-9595-b2c481b6c322 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f00f7372-2dfb-43bd-9595-b2c481b6c322', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:29,254 [235] [DEBUG] [app] Ending request: urn:request:4b3e330d-61a8-4bfe-95c1-14b39269680f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4b3e330d-61a8-4bfe-95c1-14b39269680f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:34:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:34:30,559 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:34:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:34:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:32.625945+00:00 (in 1.004378 seconds)
namespacegcworker stdout | 2025-02-07 04:34:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:31 UTC)" (scheduled at 2025-02-07 04:34:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:34:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:34:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 31, 621887), True, datetime.datetime(2025, 2, 7, 4, 34, 31, 621887), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:34:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:34:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:34:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:34:31,705 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:34:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:34:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:31.621131+00:00 (in 58.994715 seconds)
namespacegcworker stdout | 2025-02-07 04:34:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:32 UTC)" (scheduled at 2025-02-07 04:34:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:34:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:34:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:34:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:34:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:03.919286+00:00 (in 29.999528 seconds)
buildlogsarchiver stdout | 2025-02-07 04:34:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:03 UTC)" (scheduled at 2025-02-07 04:34:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:34:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 34, 33, 919940), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:34:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:34:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:34:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:34:36,035 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:34:37,114 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:56.713547+00:00 (in 18.997198 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:37 UTC)" (scheduled at 2025-02-07 04:34:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:34:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:34:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:48.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 04:34:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:48 UTC)" (scheduled at 2025-02-07 04:34:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:34:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:34:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 38, 732852), True, datetime.datetime(2025, 2, 7, 4, 34, 38, 732852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:34:38,744 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:34:38,744 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:34:38,744 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:34:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:34:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:54.423055+00:00 (in 13.998371 seconds)
gcworker stdout | 2025-02-07 04:34:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:10 UTC)" (scheduled at 2025-02-07 04:34:40.424250+00:00)
gcworker stdout | 2025-02-07 04:34:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:34:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902580431, None, 1, 0])
gcworker stdout | 2025-02-07 04:34:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:34:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:34:41,002 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:34:41,737 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:34:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:34:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:11.831521+00:00 (in 29.999566 seconds)
autopruneworker stdout | 2025-02-07 04:34:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:11 UTC)" (scheduled at 2025-02-07 04:34:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:34:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899281836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:34:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:34:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:34:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:11 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:34:42,123 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:34:43,462 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:34:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:34:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:59.225378+00:00 (in 14.999556 seconds)
quotatotalworker stdout | 2025-02-07 04:34:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:59 UTC)" (scheduled at 2025-02-07 04:34:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:34:44,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:34:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:34:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:34:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:34:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:34:44,235 [235] [DEBUG] [app] Starting request: urn:request:bd8c0534-a6bb-4be6-a691-cc130eecf445 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:44,235 [237] [DEBUG] [app] Starting request: urn:request:636ac58f-5d54-4157-a468-6448bca0da8f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:34:44,238 [239] [DEBUG] [app] Starting request: urn:request:6333b907-57df-471a-a4b6-d9678fcddc4c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:44,238 [244] [DEBUG] [app] Starting request: urn:request:f3389b15-c4ac-4036-a47d-a3852f48b810 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:44,238 [239] [DEBUG] [app] Ending request: urn:request:6333b907-57df-471a-a4b6-d9678fcddc4c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6333b907-57df-471a-a4b6-d9678fcddc4c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:44,239 [244] [DEBUG] [app] Ending request: urn:request:f3389b15-c4ac-4036-a47d-a3852f48b810 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3389b15-c4ac-4036-a47d-a3852f48b810', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:34:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:34:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:44,241 [238] [DEBUG] [app] Starting request: urn:request:470629a0-ab87-4584-bab0-c144cb2bc29a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:44,242 [238] [DEBUG] [app] Ending request: urn:request:470629a0-ab87-4584-bab0-c144cb2bc29a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:470629a0-ab87-4584-bab0-c144cb2bc29a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:34:44,242 [235] [DEBUG] [app] Starting request: urn:request:0c48f024-91e6-40ba-99b9-26a5a542bde2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:34:44,242 [235] [DEBUG] [app] Ending request: urn:request:0c48f024-91e6-40ba-99b9-26a5a542bde2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0c48f024-91e6-40ba-99b9-26a5a542bde2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:34:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:34:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:44,254 [235] [DEBUG] [app] Ending request: urn:request:bd8c0534-a6bb-4be6-a691-cc130eecf445 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd8c0534-a6bb-4be6-a691-cc130eecf445', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:44,254 [237] [DEBUG] [app] Ending request: urn:request:636ac58f-5d54-4157-a468-6448bca0da8f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:636ac58f-5d54-4157-a468-6448bca0da8f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:34:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:34:44,389 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:34:44,474 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:34:45,458 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:46,053 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:34:46,477 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:34:46,497 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:34:46,661 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:34:46,847 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:34:46,850 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:34:46,855 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:34:46,858 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:34:46,860 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:34:47,098 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:34:47,568 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:34:47,721 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:34:47,756 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:34:47,870 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:34:47,874 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:34:47,876 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:34:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:34:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:50.814677+00:00 (in 2.082071 seconds)
notificationworker stdout | 2025-02-07 04:34:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:58 UTC)" (scheduled at 2025-02-07 04:34:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:34:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:34:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 48, 732882), True, datetime.datetime(2025, 2, 7, 4, 34, 48, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:34:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:34:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:34:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:34:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:34:49,072 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:34:49,081 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:34:49,086 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:34:49,090 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:34:49,093 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:34:49,098 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:34:49,100 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:34:49,105 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:34:49,107 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:34:49,377 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:34:49,415 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:34:49,822 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:34:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:34:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:58.732147+00:00 (in 7.916973 seconds)
notificationworker stdout | 2025-02-07 04:34:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:50 UTC)" (scheduled at 2025-02-07 04:34:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:34:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:34:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:34:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:34:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:34:59.937899+00:00 (in 6.999816 seconds)
exportactionlogsworker stdout | 2025-02-07 04:34:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:39:52 UTC)" (scheduled at 2025-02-07 04:34:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:34:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 34, 52, 938354), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:34:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 34, 52, 938354), True, datetime.datetime(2025, 2, 7, 4, 34, 52, 938354), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:34:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 34, 52, 938354), True, datetime.datetime(2025, 2, 7, 4, 34, 52, 938354), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 34, 52, 938354), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:34:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:34:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:39:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:34:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:34:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:10.424250+00:00 (in 16.000748 seconds)
gcworker stdout | 2025-02-07 04:34:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:24 UTC)" (scheduled at 2025-02-07 04:34:54.423055+00:00)
gcworker stdout | 2025-02-07 04:34:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:34:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:34:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:34:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:56.118855+00:00 (in 59.999522 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:34:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:56 UTC)" (scheduled at 2025-02-07 04:34:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:34:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:34:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:34:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:37.715878+00:00 (in 41.001858 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:56 UTC)" (scheduled at 2025-02-07 04:34:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:34:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:34:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:34:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:08.732147+00:00 (in 9.999460 seconds)
notificationworker stdout | 2025-02-07 04:34:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:08 UTC)" (scheduled at 2025-02-07 04:34:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:34:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:34:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 34, 58, 732985), True, datetime.datetime(2025, 2, 7, 4, 34, 58, 732985), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:34:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:34:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:34:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:34:58,831 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:34:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:34:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:14.225378+00:00 (in 14.999561 seconds)
quotatotalworker stdout | 2025-02-07 04:34:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:14 UTC)" (scheduled at 2025-02-07 04:34:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:34:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:34:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:34:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:34:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:34:59,235 [235] [DEBUG] [app] Starting request: urn:request:d9039a06-67f5-402c-8c4d-d9abbd7068c7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:59,235 [238] [DEBUG] [app] Starting request: urn:request:452688a9-867a-469f-af85-3e1b7e97a60a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:34:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:34:59,238 [243] [DEBUG] [app] Starting request: urn:request:b0245c0d-8adb-4bf1-8190-2bb4521cc563 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:59,238 [239] [DEBUG] [app] Starting request: urn:request:ac4d62f8-1092-4c6c-a06d-5736eaf2dccf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:34:59,238 [239] [DEBUG] [app] Ending request: urn:request:ac4d62f8-1092-4c6c-a06d-5736eaf2dccf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac4d62f8-1092-4c6c-a06d-5736eaf2dccf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:59,238 [243] [DEBUG] [app] Ending request: urn:request:b0245c0d-8adb-4bf1-8190-2bb4521cc563 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b0245c0d-8adb-4bf1-8190-2bb4521cc563', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:34:59,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:34:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:34:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:34:59,241 [236] [DEBUG] [app] Starting request: urn:request:e328fa46-f354-4d99-9f47-1c215261db7e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:59,241 [235] [DEBUG] [app] Starting request: urn:request:d6fcf320-c367-4901-9c26-6f3c4c6d18b2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:34:59,241 [236] [DEBUG] [app] Ending request: urn:request:e328fa46-f354-4d99-9f47-1c215261db7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e328fa46-f354-4d99-9f47-1c215261db7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:34:59,242 [235] [DEBUG] [app] Ending request: urn:request:d6fcf320-c367-4901-9c26-6f3c4c6d18b2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d6fcf320-c367-4901-9c26-6f3c4c6d18b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:34:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:34:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:34:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:34:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:34:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:34:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:34:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:34:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:34:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:34:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:34:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:34:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:34:59,253 [235] [DEBUG] [app] Ending request: urn:request:d9039a06-67f5-402c-8c4d-d9abbd7068c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9039a06-67f5-402c-8c4d-d9abbd7068c7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:59,253 [238] [DEBUG] [app] Ending request: urn:request:452688a9-867a-469f-af85-3e1b7e97a60a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:452688a9-867a-469f-af85-3e1b7e97a60a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:34:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:34:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:34:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:34:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:34:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:34:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:34:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:04.936688+00:00 (in 4.998323 seconds)
exportactionlogsworker stdout | 2025-02-07 04:34:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:59 UTC)" (scheduled at 2025-02-07 04:34:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:34:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:34:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:35:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:35:00,578 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:35:01,737 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:35:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:35:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:23.420886+00:00 (in 20.001308 seconds)
repositorygcworker stdout | 2025-02-07 04:35:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:03 UTC)" (scheduled at 2025-02-07 04:35:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:35:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:35:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 3, 419848), True, datetime.datetime(2025, 2, 7, 4, 35, 3, 419848), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:35:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:35:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:35:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:35:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:35:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:33.919286+00:00 (in 29.999535 seconds)
buildlogsarchiver stdout | 2025-02-07 04:35:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:33 UTC)" (scheduled at 2025-02-07 04:35:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:35:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 35, 3, 919933), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:35:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:35:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:35:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:35:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:35:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:59.937899+00:00 (in 55.000777 seconds)
exportactionlogsworker stdout | 2025-02-07 04:35:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:04 UTC)" (scheduled at 2025-02-07 04:35:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:35:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:35:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 4, 937442), True, datetime.datetime(2025, 2, 7, 4, 35, 4, 937442), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:35:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:35:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:35:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:35:06,071 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:35:07,124 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:35:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:35:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:18.732147+00:00 (in 9.999491 seconds)
notificationworker stdout | 2025-02-07 04:35:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:18 UTC)" (scheduled at 2025-02-07 04:35:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:35:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:35:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 8, 732966), True, datetime.datetime(2025, 2, 7, 4, 35, 8, 732966), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:35:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:35:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:35:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:35:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:35:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:24.423055+00:00 (in 13.998359 seconds)
gcworker stdout | 2025-02-07 04:35:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:40 UTC)" (scheduled at 2025-02-07 04:35:10.424250+00:00)
gcworker stdout | 2025-02-07 04:35:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:35:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902610431, None, 1, 0])
gcworker stdout | 2025-02-07 04:35:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:35:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:35:11,030 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:35:11,772 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:35:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:35:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:41.831521+00:00 (in 29.999531 seconds)
autopruneworker stdout | 2025-02-07 04:35:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:41 UTC)" (scheduled at 2025-02-07 04:35:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:35:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899311836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:35:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:35:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:35:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:41 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:35:12,149 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:35:13,498 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:35:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:35:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:29.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 04:35:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:29 UTC)" (scheduled at 2025-02-07 04:35:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:35:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:35:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:35:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:35:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:35:14,235 [235] [DEBUG] [app] Starting request: urn:request:49d50617-0add-41b6-b8d3-ef972f75a073 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:14,235 [238] [DEBUG] [app] Starting request: urn:request:fb93919b-88c1-490e-aaf3-523c7fed2bfe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:35:14,238 [246] [DEBUG] [app] Starting request: urn:request:c1213866-674e-4027-aa1b-da42d93dcad0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:14,238 [239] [DEBUG] [app] Starting request: urn:request:036b52d1-4a80-4548-92be-53dcc7275fe4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:14,239 [239] [DEBUG] [app] Ending request: urn:request:036b52d1-4a80-4548-92be-53dcc7275fe4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:036b52d1-4a80-4548-92be-53dcc7275fe4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:35:14,239 [246] [DEBUG] [app] Ending request: urn:request:c1213866-674e-4027-aa1b-da42d93dcad0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c1213866-674e-4027-aa1b-da42d93dcad0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:35:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:35:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:14,241 [235] [DEBUG] [app] Starting request: urn:request:dcd6ce9a-da5b-4f9b-b56f-acf19478ca59 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:14,242 [236] [DEBUG] [app] Starting request: urn:request:de97b7fd-14c1-4c22-93f7-c7c45a6a6a06 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:14,242 [235] [DEBUG] [app] Ending request: urn:request:dcd6ce9a-da5b-4f9b-b56f-acf19478ca59 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dcd6ce9a-da5b-4f9b-b56f-acf19478ca59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:35:14,242 [236] [DEBUG] [app] Ending request: urn:request:de97b7fd-14c1-4c22-93f7-c7c45a6a6a06 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de97b7fd-14c1-4c22-93f7-c7c45a6a6a06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:35:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:35:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:14,254 [235] [DEBUG] [app] Ending request: urn:request:49d50617-0add-41b6-b8d3-ef972f75a073 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:49d50617-0add-41b6-b8d3-ef972f75a073', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:35:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:35:14,254 [238] [DEBUG] [app] Ending request: urn:request:fb93919b-88c1-490e-aaf3-523c7fed2bfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb93919b-88c1-490e-aaf3-523c7fed2bfe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:35:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:35:14,425 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:35:14,490 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:35:15,495 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:16,082 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:35:16,513 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:35:16,532 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:35:16,697 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:35:16,855 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:35:16,858 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:35:16,864 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:35:16,866 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:35:16,869 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:35:17,134 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:35:17,599 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:35:17,740 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:35:17,792 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:35:17,879 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:35:17,882 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:35:17,885 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:35:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:35:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:28.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 04:35:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:28 UTC)" (scheduled at 2025-02-07 04:35:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:35:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:35:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 18, 732895), True, datetime.datetime(2025, 2, 7, 4, 35, 18, 732895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:35:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:35:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:35:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:35:19,079 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:35:19,088 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:35:19,094 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:35:19,097 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:35:19,100 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:35:19,105 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:35:19,110 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:35:19,112 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:35:19,116 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:35:19,385 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:35:19,451 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:35:19,858 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:35:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:35:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:03.419110+00:00 (in 39.997735 seconds)
repositorygcworker stdout | 2025-02-07 04:35:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:23 UTC)" (scheduled at 2025-02-07 04:35:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:35:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:35:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:35:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:35:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:40.424250+00:00 (in 16.000746 seconds)
gcworker stdout | 2025-02-07 04:35:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:54 UTC)" (scheduled at 2025-02-07 04:35:24.423055+00:00)
gcworker stdout | 2025-02-07 04:35:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:35:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:35:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:35:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:35:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:38.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 04:35:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:38 UTC)" (scheduled at 2025-02-07 04:35:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:35:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:35:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 28, 732868), True, datetime.datetime(2025, 2, 7, 4, 35, 28, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:35:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:35:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:35:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:35:28,867 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:35:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:35:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:44.225378+00:00 (in 14.999518 seconds)
quotatotalworker stdout | 2025-02-07 04:35:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:44 UTC)" (scheduled at 2025-02-07 04:35:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:35:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:35:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:35:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:35:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:35:29,236 [235] [DEBUG] [app] Starting request: urn:request:a42a8667-3e1c-402e-9775-38a8d29d885c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:29,236 [237] [DEBUG] [app] Starting request: urn:request:6040a5f2-595a-4344-8817-4c7a8104ffb6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:29,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:35:29,240 [245] [DEBUG] [app] Starting request: urn:request:062a35c7-958d-4185-b99d-89f1964c1f73 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:29,240 [239] [DEBUG] [app] Starting request: urn:request:e40f3d5b-8e57-4f7a-bb7f-966ff7f4e4eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:29,240 [245] [DEBUG] [app] Ending request: urn:request:062a35c7-958d-4185-b99d-89f1964c1f73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:062a35c7-958d-4185-b99d-89f1964c1f73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:35:29,240 [239] [DEBUG] [app] Ending request: urn:request:e40f3d5b-8e57-4f7a-bb7f-966ff7f4e4eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e40f3d5b-8e57-4f7a-bb7f-966ff7f4e4eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:35:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:35:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:29,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:35:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:29,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:29,244 [236] [DEBUG] [app] Starting request: urn:request:1a811a80-455d-4678-af8a-334be61958c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:29,244 [238] [DEBUG] [app] Starting request: urn:request:4c5caa99-c87f-4a92-826e-33661de69298 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:29,244 [238] [DEBUG] [app] Ending request: urn:request:4c5caa99-c87f-4a92-826e-33661de69298 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c5caa99-c87f-4a92-826e-33661de69298', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:35:29,244 [236] [DEBUG] [app] Ending request: urn:request:1a811a80-455d-4678-af8a-334be61958c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1a811a80-455d-4678-af8a-334be61958c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:35:29,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:35:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:29,245 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:29,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:29,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:29,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:29,250 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:29,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:29,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:29,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:29,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:29,256 [235] [DEBUG] [app] Ending request: urn:request:a42a8667-3e1c-402e-9775-38a8d29d885c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a42a8667-3e1c-402e-9775-38a8d29d885c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:29,256 [237] [DEBUG] [app] Ending request: urn:request:6040a5f2-595a-4344-8817-4c7a8104ffb6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6040a5f2-595a-4344-8817-4c7a8104ffb6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:29,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:35:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:35:29,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:35:29,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
repositoryactioncounter stdout | 2025-02-07 04:35:30,601 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:35:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:35:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:32.625945+00:00 (in 1.004364 seconds)
namespacegcworker stdout | 2025-02-07 04:35:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:31 UTC)" (scheduled at 2025-02-07 04:35:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:35:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:35:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 31, 621851), True, datetime.datetime(2025, 2, 7, 4, 35, 31, 621851), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:35:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:35:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:35:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:35:31,774 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:35:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:35:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:31.621131+00:00 (in 58.994719 seconds)
namespacegcworker stdout | 2025-02-07 04:35:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:32 UTC)" (scheduled at 2025-02-07 04:35:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:35:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:35:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:35:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:35:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:03.919286+00:00 (in 29.999500 seconds)
buildlogsarchiver stdout | 2025-02-07 04:35:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:03 UTC)" (scheduled at 2025-02-07 04:35:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:35:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 35, 33, 920066), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:35:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:35:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:35:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:35:36,107 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:35:37,146 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:56.713547+00:00 (in 18.997168 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:37 UTC)" (scheduled at 2025-02-07 04:35:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:35:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:35:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:48.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 04:35:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:48 UTC)" (scheduled at 2025-02-07 04:35:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:35:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:35:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 38, 732867), True, datetime.datetime(2025, 2, 7, 4, 35, 38, 732867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:35:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:35:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:35:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:35:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:35:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:54.423055+00:00 (in 13.998393 seconds)
gcworker stdout | 2025-02-07 04:35:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:10 UTC)" (scheduled at 2025-02-07 04:35:40.424250+00:00)
gcworker stdout | 2025-02-07 04:35:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:35:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902640431, None, 1, 0])
gcworker stdout | 2025-02-07 04:35:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:35:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:35:41,038 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:35:41,786 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:35:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:35:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:11.831521+00:00 (in 29.999521 seconds)
autopruneworker stdout | 2025-02-07 04:35:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:11 UTC)" (scheduled at 2025-02-07 04:35:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:35:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899341836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:35:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:35:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:35:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:11 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:35:42,185 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:35:43,505 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:35:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:35:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:59.225378+00:00 (in 14.999543 seconds)
quotatotalworker stdout | 2025-02-07 04:35:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:59 UTC)" (scheduled at 2025-02-07 04:35:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:35:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:35:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:35:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:35:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:35:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:35:44,236 [238] [DEBUG] [app] Starting request: urn:request:63dd278a-1a5e-47b2-a375-a71ff7171e6e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:44,236 [235] [DEBUG] [app] Starting request: urn:request:38fc4c5f-07cc-49ac-a0e9-ba404dec40a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:35:44,239 [239] [DEBUG] [app] Starting request: urn:request:05da11b6-e2c1-4886-9ef4-d4d63c69a481 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:44,240 [246] [DEBUG] [app] Starting request: urn:request:7b040810-73f5-4273-84c4-e8aa9f5de4d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:44,240 [239] [DEBUG] [app] Ending request: urn:request:05da11b6-e2c1-4886-9ef4-d4d63c69a481 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:05da11b6-e2c1-4886-9ef4-d4d63c69a481', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:35:44,240 [246] [DEBUG] [app] Ending request: urn:request:7b040810-73f5-4273-84c4-e8aa9f5de4d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b040810-73f5-4273-84c4-e8aa9f5de4d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:35:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:35:44,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:44,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:44,243 [236] [DEBUG] [app] Starting request: urn:request:c9408284-3e0f-422b-8b7a-84485fcbc8fb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:44,243 [236] [DEBUG] [app] Ending request: urn:request:c9408284-3e0f-422b-8b7a-84485fcbc8fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9408284-3e0f-422b-8b7a-84485fcbc8fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:35:44,244 [238] [DEBUG] [app] Starting request: urn:request:d1364f06-b621-4b71-b377-207affaf28ce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:35:44,244 [238] [DEBUG] [app] Ending request: urn:request:d1364f06-b621-4b71-b377-207affaf28ce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d1364f06-b621-4b71-b377-207affaf28ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:35:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:35:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:44,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:44,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:44,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:44,256 [238] [DEBUG] [app] Ending request: urn:request:63dd278a-1a5e-47b2-a375-a71ff7171e6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63dd278a-1a5e-47b2-a375-a71ff7171e6e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:35:44,257 [235] [DEBUG] [app] Ending request: urn:request:38fc4c5f-07cc-49ac-a0e9-ba404dec40a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38fc4c5f-07cc-49ac-a0e9-ba404dec40a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:44,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:35:44,257 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 04:35:44,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:35:44,462 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:35:44,506 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:35:45,531 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:46,090 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:35:46,550 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:35:46,567 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:35:46,719 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:35:46,863 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:35:46,866 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:35:46,872 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:35:46,875 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:35:46,878 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:35:47,164 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:35:47,636 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:35:47,776 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:35:47,818 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:35:47,888 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:35:47,891 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:35:47,894 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:35:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:35:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:50.814677+00:00 (in 2.082097 seconds)
notificationworker stdout | 2025-02-07 04:35:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:58 UTC)" (scheduled at 2025-02-07 04:35:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:35:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:35:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 48, 732884), True, datetime.datetime(2025, 2, 7, 4, 35, 48, 732884), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:35:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:35:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:35:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:35:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:35:49,089 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:35:49,097 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:35:49,102 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:35:49,105 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:35:49,108 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:35:49,111 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:35:49,117 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:35:49,120 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:35:49,123 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:35:49,415 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:35:49,480 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:35:49,892 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:35:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:35:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:35:58.732147+00:00 (in 7.917022 seconds)
notificationworker stdout | 2025-02-07 04:35:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:50 UTC)" (scheduled at 2025-02-07 04:35:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:35:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:35:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:35:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:35:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:10.424250+00:00 (in 16.000705 seconds)
gcworker stdout | 2025-02-07 04:35:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:24 UTC)" (scheduled at 2025-02-07 04:35:54.423055+00:00)
gcworker stdout | 2025-02-07 04:35:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:35:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:35:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:35:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:56.118855+00:00 (in 59.999520 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:35:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:56 UTC)" (scheduled at 2025-02-07 04:35:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:35:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:35:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:35:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:37.715878+00:00 (in 41.001826 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:56 UTC)" (scheduled at 2025-02-07 04:35:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:35:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:35:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:35:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:08.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 04:35:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:08 UTC)" (scheduled at 2025-02-07 04:35:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:35:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:35:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 35, 58, 732850), True, datetime.datetime(2025, 2, 7, 4, 35, 58, 732850), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:35:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:35:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:35:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:35:58,887 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:35:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:35:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:14.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 04:35:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:14 UTC)" (scheduled at 2025-02-07 04:35:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:35:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:35:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:35:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:35:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:35:59,235 [235] [DEBUG] [app] Starting request: urn:request:7857955e-e17c-46c0-ae83-60701b557617 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:59,235 [237] [DEBUG] [app] Starting request: urn:request:596150c7-aca5-42e4-b1ef-b28dcc8949fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:35:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:35:59,238 [239] [DEBUG] [app] Starting request: urn:request:d9e17640-6265-4b67-86ec-ae0be977352d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:59,238 [242] [DEBUG] [app] Starting request: urn:request:0529ffa1-04b3-4543-87a6-b41dcfa81aad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:35:59,238 [242] [DEBUG] [app] Ending request: urn:request:0529ffa1-04b3-4543-87a6-b41dcfa81aad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0529ffa1-04b3-4543-87a6-b41dcfa81aad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:35:59,238 [239] [DEBUG] [app] Ending request: urn:request:d9e17640-6265-4b67-86ec-ae0be977352d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d9e17640-6265-4b67-86ec-ae0be977352d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:35:59,238 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:35:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:59,238 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:35:59,241 [238] [DEBUG] [app] Starting request: urn:request:d17bfacf-05ce-49c6-80a9-9cbe6aef69cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:59,241 [238] [DEBUG] [app] Ending request: urn:request:d17bfacf-05ce-49c6-80a9-9cbe6aef69cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d17bfacf-05ce-49c6-80a9-9cbe6aef69cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:35:59,241 [237] [DEBUG] [app] Starting request: urn:request:78526354-49b2-4aaa-aed9-6a78b32b0b3a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:35:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:35:59,242 [237] [DEBUG] [app] Ending request: urn:request:78526354-49b2-4aaa-aed9-6a78b32b0b3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78526354-49b2-4aaa-aed9-6a78b32b0b3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:35:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:35:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:35:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:35:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:35:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:35:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:35:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:35:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:35:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:35:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:35:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:35:59,253 [237] [DEBUG] [app] Ending request: urn:request:596150c7-aca5-42e4-b1ef-b28dcc8949fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:596150c7-aca5-42e4-b1ef-b28dcc8949fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:35:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:35:59,254 [235] [DEBUG] [app] Ending request: urn:request:7857955e-e17c-46c0-ae83-60701b557617 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7857955e-e17c-46c0-ae83-60701b557617', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:35:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:35:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:35:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:35:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:35:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:35:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:04.936688+00:00 (in 4.998303 seconds)
exportactionlogsworker stdout | 2025-02-07 04:35:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:59 UTC)" (scheduled at 2025-02-07 04:35:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:35:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:35:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:36:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:36:00,614 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:36:01,793 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:36:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:36:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:23.420886+00:00 (in 20.001332 seconds)
repositorygcworker stdout | 2025-02-07 04:36:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:03 UTC)" (scheduled at 2025-02-07 04:36:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:36:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:36:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 3, 419825), True, datetime.datetime(2025, 2, 7, 4, 36, 3, 419825), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:36:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:36:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:36:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:36:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:36:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:33.919286+00:00 (in 29.999523 seconds)
buildlogsarchiver stdout | 2025-02-07 04:36:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:33 UTC)" (scheduled at 2025-02-07 04:36:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:36:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 36, 3, 920009), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:36:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:36:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:36:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:36:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:36:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:59.937899+00:00 (in 55.000796 seconds)
exportactionlogsworker stdout | 2025-02-07 04:36:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:04 UTC)" (scheduled at 2025-02-07 04:36:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:36:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:36:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 4, 937438), True, datetime.datetime(2025, 2, 7, 4, 36, 4, 937438), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:36:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:36:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:36:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:36:06,141 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:36:07,182 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:36:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:36:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:18.732147+00:00 (in 9.999534 seconds)
notificationworker stdout | 2025-02-07 04:36:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:18 UTC)" (scheduled at 2025-02-07 04:36:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:36:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:36:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 8, 732934), True, datetime.datetime(2025, 2, 7, 4, 36, 8, 732934), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:36:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:36:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:36:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:36:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:36:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:24.423055+00:00 (in 13.998331 seconds)
gcworker stdout | 2025-02-07 04:36:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:40 UTC)" (scheduled at 2025-02-07 04:36:10.424250+00:00)
gcworker stdout | 2025-02-07 04:36:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:36:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902670431, None, 1, 0])
gcworker stdout | 2025-02-07 04:36:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:36:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:36:11,074 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:36:11,822 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
autopruneworker stdout | 2025-02-07 04:36:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:36:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:41.831521+00:00 (in 29.999572 seconds)
autopruneworker stdout | 2025-02-07 04:36:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:41 UTC)" (scheduled at 2025-02-07 04:36:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:36:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899371836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:36:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:36:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:36:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:41 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 04:36:12,221 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:36:13,541 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:36:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:36:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:29.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 04:36:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:29 UTC)" (scheduled at 2025-02-07 04:36:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:36:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:36:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:36:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:36:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:36:14,236 [235] [DEBUG] [app] Starting request: urn:request:70056c94-e9e1-4761-883c-c8080b23573d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:14,236 [237] [DEBUG] [app] Starting request: urn:request:d1ac4fa9-221c-40dc-973f-e6e45055ab4d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:36:14,239 [239] [DEBUG] [app] Starting request: urn:request:f3eb0dbb-9bad-46cf-8088-07ccfecd6f39 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:14,239 [246] [DEBUG] [app] Starting request: urn:request:44a63e87-800d-4773-bc19-4d048feb58f1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:14,240 [239] [DEBUG] [app] Ending request: urn:request:f3eb0dbb-9bad-46cf-8088-07ccfecd6f39 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3eb0dbb-9bad-46cf-8088-07ccfecd6f39', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:36:14,240 [246] [DEBUG] [app] Ending request: urn:request:44a63e87-800d-4773-bc19-4d048feb58f1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:44a63e87-800d-4773-bc19-4d048feb58f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:36:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:36:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:36:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:14,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:14,242 [235] [DEBUG] [app] Starting request: urn:request:0421fe67-b5c3-4092-88d1-a26dcaa3548a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:36:14,243 [235] [DEBUG] [app] Ending request: urn:request:0421fe67-b5c3-4092-88d1-a26dcaa3548a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0421fe67-b5c3-4092-88d1-a26dcaa3548a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:14,243 [238] [DEBUG] [app] Starting request: urn:request:c277d27b-d704-4905-b2f6-08925b34a89d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:36:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:36:14,243 [238] [DEBUG] [app] Ending request: urn:request:c277d27b-d704-4905-b2f6-08925b34a89d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c277d27b-d704-4905-b2f6-08925b34a89d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:36:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:36:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:36:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:36:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:14,254 [235] [DEBUG] [app] Ending request: urn:request:70056c94-e9e1-4761-883c-c8080b23573d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:70056c94-e9e1-4761-883c-c8080b23573d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:36:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:36:14,255 [237] [DEBUG] [app] Ending request: urn:request:d1ac4fa9-221c-40dc-973f-e6e45055ab4d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1ac4fa9-221c-40dc-973f-e6e45055ab4d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:36:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:36:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:36:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:36:14,498 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:36:14,530 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:36:15,568 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:16,101 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:36:16,570 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:36:16,596 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:36:16,746 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:36:16,872 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:36:16,876 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:36:16,879 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:36:16,882 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:36:16,885 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:36:17,186 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:36:17,658 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:36:17,809 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:36:17,843 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:36:17,896 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:36:17,899 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:36:17,901 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:36:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:36:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:28.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:36:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:28 UTC)" (scheduled at 2025-02-07 04:36:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:36:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:36:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 18, 732859), True, datetime.datetime(2025, 2, 7, 4, 36, 18, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:36:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:36:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:36:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:36:19,100 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:36:19,104 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:36:19,109 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:36:19,112 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:36:19,114 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:36:19,117 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:36:19,125 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:36:19,131 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:36:19,134 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:36:19,452 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:36:19,491 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:36:19,901 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:36:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:36:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:03.419110+00:00 (in 39.997763 seconds)
repositorygcworker stdout | 2025-02-07 04:36:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:23 UTC)" (scheduled at 2025-02-07 04:36:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:36:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:36:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:36:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:36:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:40.424250+00:00 (in 16.000748 seconds)
gcworker stdout | 2025-02-07 04:36:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:54 UTC)" (scheduled at 2025-02-07 04:36:24.423055+00:00)
gcworker stdout | 2025-02-07 04:36:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:36:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:36:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:36:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:36:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:38.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 04:36:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:38 UTC)" (scheduled at 2025-02-07 04:36:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:36:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:36:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 28, 732886), True, datetime.datetime(2025, 2, 7, 4, 36, 28, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:36:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:36:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:36:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:36:28,917 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:36:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:36:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:44.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 04:36:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:44 UTC)" (scheduled at 2025-02-07 04:36:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:36:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:36:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:36:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:36:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:36:29,235 [235] [DEBUG] [app] Starting request: urn:request:7cc036a1-50b3-40f9-81dd-366ecdfd4c51 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:29,235 [238] [DEBUG] [app] Starting request: urn:request:fef20306-ee35-4116-8854-3216742b5b78 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:36:29,238 [239] [DEBUG] [app] Starting request: urn:request:d6f11e68-7bed-4cce-a3ef-672a5fcd6c64 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:29,238 [246] [DEBUG] [app] Starting request: urn:request:52c31e53-d7bb-4785-990c-e893ffc07975 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:29,239 [239] [DEBUG] [app] Ending request: urn:request:d6f11e68-7bed-4cce-a3ef-672a5fcd6c64 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6f11e68-7bed-4cce-a3ef-672a5fcd6c64', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:36:29,239 [246] [DEBUG] [app] Ending request: urn:request:52c31e53-d7bb-4785-990c-e893ffc07975 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:52c31e53-d7bb-4785-990c-e893ffc07975', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:36:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:36:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:36:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:29,242 [238] [DEBUG] [app] Starting request: urn:request:0b85947d-3f1a-40fe-b9fe-d309e9087768 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:36:29,242 [238] [DEBUG] [app] Ending request: urn:request:0b85947d-3f1a-40fe-b9fe-d309e9087768 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b85947d-3f1a-40fe-b9fe-d309e9087768', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:29,242 [236] [DEBUG] [app] Starting request: urn:request:957e42d3-18b1-456f-9524-64e16d49dcb1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:36:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:36:29,243 [236] [DEBUG] [app] Ending request: urn:request:957e42d3-18b1-456f-9524-64e16d49dcb1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:957e42d3-18b1-456f-9524-64e16d49dcb1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:36:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:36:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:36:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:29,254 [238] [DEBUG] [app] Ending request: urn:request:fef20306-ee35-4116-8854-3216742b5b78 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fef20306-ee35-4116-8854-3216742b5b78', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:36:29,254 [235] [DEBUG] [app] Ending request: urn:request:7cc036a1-50b3-40f9-81dd-366ecdfd4c51 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7cc036a1-50b3-40f9-81dd-366ecdfd4c51', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:36:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:36:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:36:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:36:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:36:30,645 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:36:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:36:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:32.625945+00:00 (in 1.004343 seconds)
namespacegcworker stdout | 2025-02-07 04:36:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:31 UTC)" (scheduled at 2025-02-07 04:36:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:36:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:36:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 31, 621883), True, datetime.datetime(2025, 2, 7, 4, 36, 31, 621883), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:36:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:36:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:36:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:36:31,829 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:36:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:36:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:13.624301+00:00 (in 40.997908 seconds)
namespacegcworker stdout | 2025-02-07 04:36:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:32 UTC)" (scheduled at 2025-02-07 04:36:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:36:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:36:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:36:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:36:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:03.919286+00:00 (in 29.999536 seconds)
buildlogsarchiver stdout | 2025-02-07 04:36:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:03 UTC)" (scheduled at 2025-02-07 04:36:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:36:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 36, 33, 920007), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:36:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:36:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:36:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:36:36,176 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:36:37,205 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:56.713547+00:00 (in 18.997187 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:37 UTC)" (scheduled at 2025-02-07 04:36:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:36:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:36:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:48.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 04:36:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:48 UTC)" (scheduled at 2025-02-07 04:36:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:36:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:36:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 38, 732804), True, datetime.datetime(2025, 2, 7, 4, 36, 38, 732804), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:36:38,745 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:36:38,745 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:36:38,745 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:36:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:36:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:54.423055+00:00 (in 13.998356 seconds)
gcworker stdout | 2025-02-07 04:36:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:10 UTC)" (scheduled at 2025-02-07 04:36:40.424250+00:00)
gcworker stdout | 2025-02-07 04:36:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:36:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902700431, None, 1, 0])
gcworker stdout | 2025-02-07 04:36:40,434 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:36:40,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:36:41,110 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:36:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:36:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:11.831521+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2025-02-07 04:36:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:11 UTC)" (scheduled at 2025-02-07 04:36:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:36:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899401836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:36:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:36:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:36:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:36:41,854 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:36:42,241 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:36:43,577 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:36:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:36:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:59.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 04:36:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:59 UTC)" (scheduled at 2025-02-07 04:36:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:36:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:36:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:36:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:36:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:36:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:36:44,235 [236] [DEBUG] [app] Starting request: urn:request:20a5edc5-5219-4d45-bd0c-d7216fa9296d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:44,235 [237] [DEBUG] [app] Starting request: urn:request:ec195ae9-0648-469e-8269-fcadd634e97f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:36:44,238 [239] [DEBUG] [app] Starting request: urn:request:abe0a6e6-8d6f-4e86-a891-429205afa5be (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:44,238 [245] [DEBUG] [app] Starting request: urn:request:0b6fef42-09c2-4fce-ba76-56c56de9d671 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:44,238 [239] [DEBUG] [app] Ending request: urn:request:abe0a6e6-8d6f-4e86-a891-429205afa5be (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abe0a6e6-8d6f-4e86-a891-429205afa5be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:36:44,239 [245] [DEBUG] [app] Ending request: urn:request:0b6fef42-09c2-4fce-ba76-56c56de9d671 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b6fef42-09c2-4fce-ba76-56c56de9d671', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:36:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:36:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:36:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:36:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:44,242 [237] [DEBUG] [app] Starting request: urn:request:f2514a88-cce7-4d4f-9bac-9d526ced7d1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:36:44,242 [237] [DEBUG] [app] Ending request: urn:request:f2514a88-cce7-4d4f-9bac-9d526ced7d1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2514a88-cce7-4d4f-9bac-9d526ced7d1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:44,242 [235] [DEBUG] [app] Starting request: urn:request:5e83a7b5-a590-4df6-91e7-b3e80e261340 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:36:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:36:44,242 [235] [DEBUG] [app] Ending request: urn:request:5e83a7b5-a590-4df6-91e7-b3e80e261340 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5e83a7b5-a590-4df6-91e7-b3e80e261340', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:36:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:36:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:36:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:44,253 [237] [DEBUG] [app] Ending request: urn:request:ec195ae9-0648-469e-8269-fcadd634e97f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec195ae9-0648-469e-8269-fcadd634e97f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:36:44,253 [236] [DEBUG] [app] Ending request: urn:request:20a5edc5-5219-4d45-bd0c-d7216fa9296d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20a5edc5-5219-4d45-bd0c-d7216fa9296d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:36:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:36:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:36:44,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:36:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:36:44,534 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:36:44,566 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:36:45,578 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:46,138 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:36:46,607 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:36:46,632 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:36:46,783 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:36:46,882 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:36:46,885 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:36:46,888 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:36:46,891 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:36:46,893 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:36:47,222 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:36:47,695 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:36:47,845 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:36:47,879 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:36:47,904 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:36:47,907 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:36:47,909 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:36:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:36:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:50.814677+00:00 (in 2.082022 seconds)
notificationworker stdout | 2025-02-07 04:36:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:58 UTC)" (scheduled at 2025-02-07 04:36:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:36:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:36:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 48, 732986), True, datetime.datetime(2025, 2, 7, 4, 36, 48, 732986), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:36:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:36:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:36:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:36:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:36:49,110 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:36:49,113 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:36:49,121 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:36:49,124 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:36:49,127 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:36:49,129 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:36:49,136 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:36:49,143 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:36:49,146 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:36:49,458 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:36:49,528 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:36:49,932 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:36:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:36:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:36:58.732147+00:00 (in 7.916985 seconds)
notificationworker stdout | 2025-02-07 04:36:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:50 UTC)" (scheduled at 2025-02-07 04:36:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:36:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:36:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:36:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:36:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:10.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 04:36:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:24 UTC)" (scheduled at 2025-02-07 04:36:54.423055+00:00)
gcworker stdout | 2025-02-07 04:36:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:36:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:36:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:36:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:56.118855+00:00 (in 59.999491 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:36:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:56 UTC)" (scheduled at 2025-02-07 04:36:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:36:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:36:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:36:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:37.715878+00:00 (in 41.001828 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:56 UTC)" (scheduled at 2025-02-07 04:36:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:36:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:36:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:36:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:08.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 04:36:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:08 UTC)" (scheduled at 2025-02-07 04:36:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:36:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:36:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 36, 58, 732880), True, datetime.datetime(2025, 2, 7, 4, 36, 58, 732880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:36:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:36:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:36:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:36:58,953 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:36:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:36:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:14.225378+00:00 (in 14.999579 seconds)
quotatotalworker stdout | 2025-02-07 04:36:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:14 UTC)" (scheduled at 2025-02-07 04:36:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:36:59,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:36:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:36:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:36:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:36:59,235 [235] [DEBUG] [app] Starting request: urn:request:07abe14c-209b-4adc-a609-f2e0c80c62a2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:59,235 [238] [DEBUG] [app] Starting request: urn:request:8191272e-1296-4b68-bb55-facf34aed9bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:36:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:36:59,238 [245] [DEBUG] [app] Starting request: urn:request:c581c8c0-5a00-4c61-ade9-97d3315c2de2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:59,238 [239] [DEBUG] [app] Starting request: urn:request:121c298c-ad9e-4e7a-b592-3e6c093172a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:36:59,239 [239] [DEBUG] [app] Ending request: urn:request:121c298c-ad9e-4e7a-b592-3e6c093172a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:121c298c-ad9e-4e7a-b592-3e6c093172a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:36:59,239 [245] [DEBUG] [app] Ending request: urn:request:c581c8c0-5a00-4c61-ade9-97d3315c2de2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c581c8c0-5a00-4c61-ade9-97d3315c2de2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:36:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:36:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:36:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:36:59,241 [235] [DEBUG] [app] Starting request: urn:request:3e1137a4-e59f-40e5-9c9a-6bcb203a32bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:36:59,242 [235] [DEBUG] [app] Ending request: urn:request:3e1137a4-e59f-40e5-9c9a-6bcb203a32bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e1137a4-e59f-40e5-9c9a-6bcb203a32bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:36:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:59,242 [236] [DEBUG] [app] Starting request: urn:request:4b37f408-85ff-4de8-bd8c-35a55ca6d2e6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:36:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:36:59,243 [236] [DEBUG] [app] Ending request: urn:request:4b37f408-85ff-4de8-bd8c-35a55ca6d2e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b37f408-85ff-4de8-bd8c-35a55ca6d2e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:36:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:36:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:36:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:36:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:36:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:36:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:36:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:36:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:36:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:36:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:36:59,253 [235] [DEBUG] [app] Ending request: urn:request:07abe14c-209b-4adc-a609-f2e0c80c62a2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07abe14c-209b-4adc-a609-f2e0c80c62a2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:36:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:36:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:36:59,254 [238] [DEBUG] [app] Ending request: urn:request:8191272e-1296-4b68-bb55-facf34aed9bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8191272e-1296-4b68-bb55-facf34aed9bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:36:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:36:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:36:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:36:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:36:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:36:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:04.936688+00:00 (in 4.998312 seconds)
exportactionlogsworker stdout | 2025-02-07 04:36:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:59 UTC)" (scheduled at 2025-02-07 04:36:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:36:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:36:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:37:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:37:00,681 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:37:01,843 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:37:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:37:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:23.420886+00:00 (in 20.001326 seconds)
repositorygcworker stdout | 2025-02-07 04:37:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:03 UTC)" (scheduled at 2025-02-07 04:37:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:37:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:37:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 3, 419830), True, datetime.datetime(2025, 2, 7, 4, 37, 3, 419830), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:37:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:37:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:37:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:37:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:37:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:33.919286+00:00 (in 29.999574 seconds)
buildlogsarchiver stdout | 2025-02-07 04:37:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:33 UTC)" (scheduled at 2025-02-07 04:37:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:37:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 37, 3, 919965), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:37:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:37:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:37:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:37:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:37:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:59.937899+00:00 (in 55.000751 seconds)
exportactionlogsworker stdout | 2025-02-07 04:37:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:04 UTC)" (scheduled at 2025-02-07 04:37:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:37:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:37:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 4, 937437), True, datetime.datetime(2025, 2, 7, 4, 37, 4, 937437), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:37:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:37:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:37:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:37:06,212 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:37:07,234 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:37:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:18.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 04:37:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:18 UTC)" (scheduled at 2025-02-07 04:37:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:37:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:37:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 8, 732877), True, datetime.datetime(2025, 2, 7, 4, 37, 8, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:37:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:37:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:37:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:37:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:37:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:24.423055+00:00 (in 13.998383 seconds)
gcworker stdout | 2025-02-07 04:37:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:40 UTC)" (scheduled at 2025-02-07 04:37:10.424250+00:00)
gcworker stdout | 2025-02-07 04:37:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:37:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902730431, None, 1, 0])
gcworker stdout | 2025-02-07 04:37:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:37:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:37:11,146 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:37:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:37:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:41.831521+00:00 (in 29.999560 seconds)
autopruneworker stdout | 2025-02-07 04:37:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:41 UTC)" (scheduled at 2025-02-07 04:37:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:37:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899431836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:37:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:37:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:37:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:37:11,861 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:37:12,277 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:37:13,614 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
namespacegcworker stdout | 2025-02-07 04:37:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:37:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:31.621131+00:00 (in 17.996408 seconds)
namespacegcworker stdout | 2025-02-07 04:37:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:42:13 UTC)" (scheduled at 2025-02-07 04:37:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:37:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 37, 13, 624954), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:37:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 37, 13, 624954), True, datetime.datetime(2025, 2, 7, 4, 37, 13, 624954), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:37:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 37, 13, 624954), True, datetime.datetime(2025, 2, 7, 4, 37, 13, 624954), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 37, 13, 624954), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:37:13,636 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:37:13,636 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:42:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:37:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:37:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:29.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 04:37:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:29 UTC)" (scheduled at 2025-02-07 04:37:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:37:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:37:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:37:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:37:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:37:14,235 [235] [DEBUG] [app] Starting request: urn:request:df3f693e-319e-40cd-8e81-3e68e3366826 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:14,235 [238] [DEBUG] [app] Starting request: urn:request:d1c75cf3-68f9-4051-883a-486da2e5cca2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:37:14,238 [239] [DEBUG] [app] Starting request: urn:request:ea9e6ce6-840c-4e8e-a6e6-f9c825e2ae28 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:14,238 [246] [DEBUG] [app] Starting request: urn:request:8fdb103c-36e6-48dc-9224-c8d2abf8408b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:14,238 [239] [DEBUG] [app] Ending request: urn:request:ea9e6ce6-840c-4e8e-a6e6-f9c825e2ae28 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea9e6ce6-840c-4e8e-a6e6-f9c825e2ae28', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:14,238 [246] [DEBUG] [app] Ending request: urn:request:8fdb103c-36e6-48dc-9224-c8d2abf8408b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fdb103c-36e6-48dc-9224-c8d2abf8408b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:37:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:37:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:14,241 [235] [DEBUG] [app] Starting request: urn:request:c28a0ab2-7c43-4047-bf1d-480ad72a65d0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:37:14,241 [235] [DEBUG] [app] Ending request: urn:request:c28a0ab2-7c43-4047-bf1d-480ad72a65d0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c28a0ab2-7c43-4047-bf1d-480ad72a65d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:37:14,241 [236] [DEBUG] [app] Starting request: urn:request:f2c84f90-3a93-4268-880e-f663d86acac3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:37:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:37:14,242 [236] [DEBUG] [app] Ending request: urn:request:f2c84f90-3a93-4268-880e-f663d86acac3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2c84f90-3a93-4268-880e-f663d86acac3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:37:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:37:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:37:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:14,253 [235] [DEBUG] [app] Ending request: urn:request:df3f693e-319e-40cd-8e81-3e68e3366826 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df3f693e-319e-40cd-8e81-3e68e3366826', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:14,253 [238] [DEBUG] [app] Ending request: urn:request:d1c75cf3-68f9-4051-883a-486da2e5cca2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1c75cf3-68f9-4051-883a-486da2e5cca2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:37:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:37:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:37:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
notificationworker stdout | 2025-02-07 04:37:14,552 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:37:14,602 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:37:15,610 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:16,162 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:37:16,641 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:37:16,653 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:37:16,797 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:37:16,890 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:37:16,893 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:37:16,896 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:37:16,899 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:37:16,903 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:37:17,258 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:37:17,731 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:37:17,882 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:37:17,912 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-secscan stdout | 2025-02-07 04:37:17,915 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:37:17,919 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:37:17,922 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 04:37:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:28.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 04:37:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:28 UTC)" (scheduled at 2025-02-07 04:37:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:37:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:37:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 18, 732819), True, datetime.datetime(2025, 2, 7, 4, 37, 18, 732819), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:37:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:37:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:37:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:37:19,119 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:37:19,122 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:37:19,131 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:37:19,136 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:37:19,139 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:37:19,141 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:37:19,146 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:37:19,153 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:37:19,156 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:37:19,490 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:37:19,542 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:37:19,969 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:37:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:37:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:03.419110+00:00 (in 39.997737 seconds)
repositorygcworker stdout | 2025-02-07 04:37:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:23 UTC)" (scheduled at 2025-02-07 04:37:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:37:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:37:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:37:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:37:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:40.424250+00:00 (in 16.000763 seconds)
gcworker stdout | 2025-02-07 04:37:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:54 UTC)" (scheduled at 2025-02-07 04:37:24.423055+00:00)
gcworker stdout | 2025-02-07 04:37:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:37:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:37:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:37:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:33.814166+00:00 (in 5.081532 seconds)
notificationworker stdout | 2025-02-07 04:37:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:38 UTC)" (scheduled at 2025-02-07 04:37:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:37:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:37:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 28, 732849), True, datetime.datetime(2025, 2, 7, 4, 37, 28, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:37:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:37:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:37:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:37:28,970 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:37:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:37:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:44.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 04:37:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:44 UTC)" (scheduled at 2025-02-07 04:37:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:37:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:37:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:37:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:37:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:37:29,236 [237] [DEBUG] [app] Starting request: urn:request:1d0b2dee-995c-4d37-b92f-43cd69438f02 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:29,236 [236] [DEBUG] [app] Starting request: urn:request:4b0d1af4-6b44-45f8-9a2f-62aca5661fbc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:37:29,239 [239] [DEBUG] [app] Starting request: urn:request:cf0141e1-854f-4e0e-a23b-cdb2271a333a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:29,239 [246] [DEBUG] [app] Starting request: urn:request:bfbf0d81-51b8-437b-92e9-cf79979a739d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:29,239 [239] [DEBUG] [app] Ending request: urn:request:cf0141e1-854f-4e0e-a23b-cdb2271a333a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf0141e1-854f-4e0e-a23b-cdb2271a333a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:29,239 [246] [DEBUG] [app] Ending request: urn:request:bfbf0d81-51b8-437b-92e9-cf79979a739d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bfbf0d81-51b8-437b-92e9-cf79979a739d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:37:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:37:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:29,242 [237] [DEBUG] [app] Starting request: urn:request:380d53c7-9b12-4dbb-9a7a-2054d3abbc51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:37:29,242 [236] [DEBUG] [app] Starting request: urn:request:b19cfc0c-4664-4cd7-871b-91ed1792e686 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:37:29,243 [237] [DEBUG] [app] Ending request: urn:request:380d53c7-9b12-4dbb-9a7a-2054d3abbc51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:380d53c7-9b12-4dbb-9a7a-2054d3abbc51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:37:29,243 [236] [DEBUG] [app] Ending request: urn:request:b19cfc0c-4664-4cd7-871b-91ed1792e686 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b19cfc0c-4664-4cd7-871b-91ed1792e686', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:37:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:37:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:37:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:29,255 [236] [DEBUG] [app] Ending request: urn:request:4b0d1af4-6b44-45f8-9a2f-62aca5661fbc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4b0d1af4-6b44-45f8-9a2f-62aca5661fbc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:29,255 [237] [DEBUG] [app] Ending request: urn:request:1d0b2dee-995c-4d37-b92f-43cd69438f02 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1d0b2dee-995c-4d37-b92f-43cd69438f02', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:37:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:37:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:37:29,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:37:30,717 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:37:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:37:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:32.625945+00:00 (in 1.004347 seconds)
namespacegcworker stdout | 2025-02-07 04:37:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:31 UTC)" (scheduled at 2025-02-07 04:37:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:37:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:37:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 31, 621864), True, datetime.datetime(2025, 2, 7, 4, 37, 31, 621864), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:37:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:37:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:37:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:37:31,850 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:37:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:37:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:31.621131+00:00 (in 58.994719 seconds)
namespacegcworker stdout | 2025-02-07 04:37:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:32 UTC)" (scheduled at 2025-02-07 04:37:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:37:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:37:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:37:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:38.732147+00:00 (in 4.917517 seconds)
notificationworker stdout | 2025-02-07 04:37:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:42:33 UTC)" (scheduled at 2025-02-07 04:37:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:37:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 37, 33, 814814), 'notification/%'])
notificationworker stdout | 2025-02-07 04:37:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 37, 33, 814814), True, datetime.datetime(2025, 2, 7, 4, 37, 33, 814814), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:37:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 37, 33, 814814), True, datetime.datetime(2025, 2, 7, 4, 37, 33, 814814), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 37, 33, 814814), 'notification/%'])
notificationworker stdout | 2025-02-07 04:37:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:37:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:42:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:37:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:37:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:03.919286+00:00 (in 29.999562 seconds)
buildlogsarchiver stdout | 2025-02-07 04:37:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:03 UTC)" (scheduled at 2025-02-07 04:37:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:37:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 37, 33, 919967), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:37:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:37:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:37:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:37:36,242 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:37:37,270 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:56.713547+00:00 (in 18.997124 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:37 UTC)" (scheduled at 2025-02-07 04:37:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:37:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:48.732147+00:00 (in 9.999534 seconds)
notificationworker stdout | 2025-02-07 04:37:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:48 UTC)" (scheduled at 2025-02-07 04:37:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:37:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:37:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 38, 732834), True, datetime.datetime(2025, 2, 7, 4, 37, 38, 732834), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:37:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:37:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:37:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:37:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:37:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:54.423055+00:00 (in 13.998358 seconds)
gcworker stdout | 2025-02-07 04:37:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:10 UTC)" (scheduled at 2025-02-07 04:37:40.424250+00:00)
gcworker stdout | 2025-02-07 04:37:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:37:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902760430, None, 1, 0])
gcworker stdout | 2025-02-07 04:37:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:37:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:37:41,184 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:37:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:37:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:11.831521+00:00 (in 29.999501 seconds)
autopruneworker stdout | 2025-02-07 04:37:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:11 UTC)" (scheduled at 2025-02-07 04:37:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:37:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899461836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:37:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:37:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:37:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:37:41,897 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:37:42,313 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:37:43,629 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:37:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:37:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:59.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 04:37:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:59 UTC)" (scheduled at 2025-02-07 04:37:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:37:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:37:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:37:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:37:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:37:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:37:44,236 [235] [DEBUG] [app] Starting request: urn:request:404a22b3-4737-4d43-beff-4519ef8908c7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:44,236 [237] [DEBUG] [app] Starting request: urn:request:f0ff7a33-153d-4aa6-a805-d1d5373ad758 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:37:44,239 [245] [DEBUG] [app] Starting request: urn:request:b7f192f5-7ee9-4061-a2d1-fbc236b1724b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:44,239 [246] [DEBUG] [app] Starting request: urn:request:dd520a3b-26ad-4edf-a02a-08e3946bc360 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:44,239 [245] [DEBUG] [app] Ending request: urn:request:b7f192f5-7ee9-4061-a2d1-fbc236b1724b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b7f192f5-7ee9-4061-a2d1-fbc236b1724b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:44,239 [246] [DEBUG] [app] Ending request: urn:request:dd520a3b-26ad-4edf-a02a-08e3946bc360 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dd520a3b-26ad-4edf-a02a-08e3946bc360', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:37:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:37:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:44,243 [238] [DEBUG] [app] Starting request: urn:request:e13fb152-ef65-48ed-b5e8-66ca0cdf3e75 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:37:44,243 [235] [DEBUG] [app] Starting request: urn:request:2addf6d5-221f-40a2-a67a-7f0b2ce5855f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:37:44,243 [238] [DEBUG] [app] Ending request: urn:request:e13fb152-ef65-48ed-b5e8-66ca0cdf3e75 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e13fb152-ef65-48ed-b5e8-66ca0cdf3e75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:37:44,243 [235] [DEBUG] [app] Ending request: urn:request:2addf6d5-221f-40a2-a67a-7f0b2ce5855f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2addf6d5-221f-40a2-a67a-7f0b2ce5855f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:37:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:37:44,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:37:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:37:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:44,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:44,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:44,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:44,256 [235] [DEBUG] [app] Ending request: urn:request:404a22b3-4737-4d43-beff-4519ef8908c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:404a22b3-4737-4d43-beff-4519ef8908c7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:44,256 [237] [DEBUG] [app] Ending request: urn:request:f0ff7a33-153d-4aa6-a805-d1d5373ad758 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0ff7a33-153d-4aa6-a805-d1d5373ad758', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:37:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:37:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:37:44,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
notificationworker stdout | 2025-02-07 04:37:44,582 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:37:44,638 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:37:45,647 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:46,199 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:37:46,677 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:37:46,681 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:37:46,831 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:37:46,897 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:37:46,900 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:37:46,903 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:37:46,906 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:37:46,909 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:37:47,297 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:37:47,767 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:37:47,919 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-secscan stdout | 2025-02-07 04:37:47,922 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:37:47,925 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:37:47,928 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestbackfillworker stdout | 2025-02-07 04:37:47,948 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:37:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:50.814677+00:00 (in 2.082077 seconds)
notificationworker stdout | 2025-02-07 04:37:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:58 UTC)" (scheduled at 2025-02-07 04:37:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:37:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:37:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 48, 732807), True, datetime.datetime(2025, 2, 7, 4, 37, 48, 732807), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:37:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:37:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:37:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:37:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:37:49,128 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:37:49,131 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:37:49,140 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:37:49,147 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:37:49,150 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:37:49,153 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:37:49,156 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:37:49,161 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:37:49,164 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:37:49,506 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:37:49,571 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:37:49,980 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:37:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:37:58.732147+00:00 (in 7.916987 seconds)
notificationworker stdout | 2025-02-07 04:37:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:50 UTC)" (scheduled at 2025-02-07 04:37:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:37:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:37:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:37:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:37:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:10.424250+00:00 (in 16.000753 seconds)
gcworker stdout | 2025-02-07 04:37:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:24 UTC)" (scheduled at 2025-02-07 04:37:54.423055+00:00)
gcworker stdout | 2025-02-07 04:37:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:37:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:37:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:37:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:56.118855+00:00 (in 59.999440 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:37:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:56 UTC)" (scheduled at 2025-02-07 04:37:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:37:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:37:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:37:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:37.715878+00:00 (in 41.001820 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:56 UTC)" (scheduled at 2025-02-07 04:37:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:37:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:37:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:37:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:08.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 04:37:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:08 UTC)" (scheduled at 2025-02-07 04:37:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:37:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:37:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 37, 58, 732888), True, datetime.datetime(2025, 2, 7, 4, 37, 58, 732888), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:37:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:37:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:37:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:37:58,980 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:37:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:37:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:14.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 04:37:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:14 UTC)" (scheduled at 2025-02-07 04:37:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:37:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:37:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:37:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:37:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:37:59,235 [236] [DEBUG] [app] Starting request: urn:request:11756574-41ae-476b-ba59-27d1c7da52d4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:59,235 [238] [DEBUG] [app] Starting request: urn:request:dae3ca26-3504-4ef3-b25f-323b983be765 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:37:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:37:59,238 [239] [DEBUG] [app] Starting request: urn:request:da94f9a0-c6cd-4f81-93ef-c1313ac563c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:59,238 [246] [DEBUG] [app] Starting request: urn:request:22b93d68-79b2-41d4-8992-6b91e3675a72 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:37:59,239 [239] [DEBUG] [app] Ending request: urn:request:da94f9a0-c6cd-4f81-93ef-c1313ac563c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da94f9a0-c6cd-4f81-93ef-c1313ac563c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:59,239 [246] [DEBUG] [app] Ending request: urn:request:22b93d68-79b2-41d4-8992-6b91e3675a72 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:22b93d68-79b2-41d4-8992-6b91e3675a72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:37:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:37:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:37:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:37:59,243 [235] [DEBUG] [app] Starting request: urn:request:5c08fa14-9664-41a5-970e-926ab2074019 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:37:59,243 [237] [DEBUG] [app] Starting request: urn:request:20cac732-ee06-4247-ade4-f57a7678cfbe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:37:59,243 [237] [DEBUG] [app] Ending request: urn:request:20cac732-ee06-4247-ade4-f57a7678cfbe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:20cac732-ee06-4247-ade4-f57a7678cfbe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:37:59,243 [235] [DEBUG] [app] Ending request: urn:request:5c08fa14-9664-41a5-970e-926ab2074019 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5c08fa14-9664-41a5-970e-926ab2074019', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:37:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:37:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:37:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:37:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:37:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:37:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:37:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:37:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:37:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:37:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:59,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:37:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:37:59,255 [238] [DEBUG] [app] Ending request: urn:request:dae3ca26-3504-4ef3-b25f-323b983be765 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dae3ca26-3504-4ef3-b25f-323b983be765', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:37:59,256 [236] [DEBUG] [app] Ending request: urn:request:11756574-41ae-476b-ba59-27d1c7da52d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11756574-41ae-476b-ba59-27d1c7da52d4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:37:59,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:37:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:37:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:37:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:37:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:37:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:37:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:04.936688+00:00 (in 4.998285 seconds)
exportactionlogsworker stdout | 2025-02-07 04:37:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:59 UTC)" (scheduled at 2025-02-07 04:37:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:37:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:37:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:38:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:38:00,744 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:38:01,886 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:38:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:38:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:23.420886+00:00 (in 20.001308 seconds)
repositorygcworker stdout | 2025-02-07 04:38:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:03 UTC)" (scheduled at 2025-02-07 04:38:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:38:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:38:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 3, 419850), True, datetime.datetime(2025, 2, 7, 4, 38, 3, 419850), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:38:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:38:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:38:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:38:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:38:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:33.919286+00:00 (in 29.999549 seconds)
buildlogsarchiver stdout | 2025-02-07 04:38:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:33 UTC)" (scheduled at 2025-02-07 04:38:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:38:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 38, 3, 919985), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:38:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:38:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:38:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:38:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:38:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:59.937899+00:00 (in 55.000787 seconds)
exportactionlogsworker stdout | 2025-02-07 04:38:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:04 UTC)" (scheduled at 2025-02-07 04:38:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:38:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:38:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 4, 937410), True, datetime.datetime(2025, 2, 7, 4, 38, 4, 937410), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:38:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:38:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:38:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:38:06,258 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:38:07,307 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:38:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:38:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:18.732147+00:00 (in 9.999521 seconds)
notificationworker stdout | 2025-02-07 04:38:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:18 UTC)" (scheduled at 2025-02-07 04:38:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:38:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:38:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 8, 732932), True, datetime.datetime(2025, 2, 7, 4, 38, 8, 732932), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:38:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:38:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:38:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:38:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:38:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:24.423055+00:00 (in 13.998340 seconds)
gcworker stdout | 2025-02-07 04:38:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:40 UTC)" (scheduled at 2025-02-07 04:38:10.424250+00:00)
gcworker stdout | 2025-02-07 04:38:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:38:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902790431, None, 1, 0])
gcworker stdout | 2025-02-07 04:38:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:38:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:38:11,210 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:38:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:38:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:41.831521+00:00 (in 29.999514 seconds)
autopruneworker stdout | 2025-02-07 04:38:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:41 UTC)" (scheduled at 2025-02-07 04:38:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:38:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899491836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:38:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:38:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:38:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:38:11,915 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:38:12,346 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:38:13,665 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:38:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:38:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:29.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 04:38:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:29 UTC)" (scheduled at 2025-02-07 04:38:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:38:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:38:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:38:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:38:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:38:14,236 [237] [DEBUG] [app] Starting request: urn:request:1042bbaf-f961-4c45-90b4-df26ead78805 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:14,236 [238] [DEBUG] [app] Starting request: urn:request:6915eed4-e102-4c04-aab4-2534fadf980e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:38:14,239 [239] [DEBUG] [app] Starting request: urn:request:4820399f-ba3b-4448-a6b4-bbf933e97c7a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:14,239 [244] [DEBUG] [app] Starting request: urn:request:587e00ff-20c6-411a-8765-c7bc932bc2c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:14,239 [244] [DEBUG] [app] Ending request: urn:request:587e00ff-20c6-411a-8765-c7bc932bc2c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:587e00ff-20c6-411a-8765-c7bc932bc2c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:38:14,239 [239] [DEBUG] [app] Ending request: urn:request:4820399f-ba3b-4448-a6b4-bbf933e97c7a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4820399f-ba3b-4448-a6b4-bbf933e97c7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:38:14,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:38:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:38:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:38:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:14,242 [235] [DEBUG] [app] Starting request: urn:request:2ca162e3-5e6e-404d-af97-fcd77d772587 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:14,242 [235] [DEBUG] [app] Ending request: urn:request:2ca162e3-5e6e-404d-af97-fcd77d772587 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2ca162e3-5e6e-404d-af97-fcd77d772587', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:38:14,243 [238] [DEBUG] [app] Starting request: urn:request:1ae047b0-c946-4fdc-a479-857602681efc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:38:14,243 [238] [DEBUG] [app] Ending request: urn:request:1ae047b0-c946-4fdc-a479-857602681efc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1ae047b0-c946-4fdc-a479-857602681efc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:38:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:38:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:38:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:38:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:14,254 [238] [DEBUG] [app] Ending request: urn:request:6915eed4-e102-4c04-aab4-2534fadf980e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6915eed4-e102-4c04-aab4-2534fadf980e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:38:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:38:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:38:14,255 [237] [DEBUG] [app] Ending request: urn:request:1042bbaf-f961-4c45-90b4-df26ead78805 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1042bbaf-f961-4c45-90b4-df26ead78805', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:38:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:38:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:38:14,610 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:38:14,674 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:38:15,683 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:16,234 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:38:16,706 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:38:16,717 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:38:16,855 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:38:16,907 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:38:16,910 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:38:16,913 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:38:16,916 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:38:16,918 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:38:17,331 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:38:17,796 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:38:17,930 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:38:17,933 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:38:17,935 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:38:17,954 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:38:17,976 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:38:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:38:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:28.732147+00:00 (in 9.999466 seconds)
notificationworker stdout | 2025-02-07 04:38:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:28 UTC)" (scheduled at 2025-02-07 04:38:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:38:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:38:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 18, 733002), True, datetime.datetime(2025, 2, 7, 4, 38, 18, 733002), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:38:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:38:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:38:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:38:19,136 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:38:19,143 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:38:19,150 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:38:19,155 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:38:19,158 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:38:19,161 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:38:19,164 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:38:19,169 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:38:19,172 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:38:19,533 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:38:19,607 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:38:20,011 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:38:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:38:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:32.420707+00:00 (in 8.999332 seconds)
repositorygcworker stdout | 2025-02-07 04:38:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:23 UTC)" (scheduled at 2025-02-07 04:38:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:38:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:38:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:38:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:38:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:40.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 04:38:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:54 UTC)" (scheduled at 2025-02-07 04:38:24.423055+00:00)
gcworker stdout | 2025-02-07 04:38:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:38:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:38:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:38:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:38:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:38.732147+00:00 (in 9.999519 seconds)
notificationworker stdout | 2025-02-07 04:38:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:38 UTC)" (scheduled at 2025-02-07 04:38:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:38:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:38:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 28, 732904), True, datetime.datetime(2025, 2, 7, 4, 38, 28, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:38:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:38:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:38:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:38:29,012 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:38:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:38:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:44.225378+00:00 (in 14.999539 seconds)
quotatotalworker stdout | 2025-02-07 04:38:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:44 UTC)" (scheduled at 2025-02-07 04:38:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:38:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:38:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:38:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:38:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:38:29,236 [238] [DEBUG] [app] Starting request: urn:request:56a53552-df60-4db6-b3b1-bca131a91e5c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:29,236 [237] [DEBUG] [app] Starting request: urn:request:a5195649-4d73-4900-adc8-f47049f18af1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:38:29,239 [239] [DEBUG] [app] Starting request: urn:request:00738f11-0140-48a5-986c-824011e976d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:29,239 [246] [DEBUG] [app] Starting request: urn:request:d7de7915-32d2-4292-89cb-9c10ffa31dec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:29,239 [246] [DEBUG] [app] Ending request: urn:request:d7de7915-32d2-4292-89cb-9c10ffa31dec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d7de7915-32d2-4292-89cb-9c10ffa31dec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:38:29,239 [239] [DEBUG] [app] Ending request: urn:request:00738f11-0140-48a5-986c-824011e976d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:00738f11-0140-48a5-986c-824011e976d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:38:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:38:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:38:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:29,242 [237] [DEBUG] [app] Starting request: urn:request:223b7ba4-08fc-4ff1-9d37-eb823c4c2366 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:29,243 [237] [DEBUG] [app] Ending request: urn:request:223b7ba4-08fc-4ff1-9d37-eb823c4c2366 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:223b7ba4-08fc-4ff1-9d37-eb823c4c2366', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:38:29,243 [238] [DEBUG] [app] Starting request: urn:request:e0050199-912f-4094-9367-fa2caa59d74c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:29,243 [238] [DEBUG] [app] Ending request: urn:request:e0050199-912f-4094-9367-fa2caa59d74c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0050199-912f-4094-9367-fa2caa59d74c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:38:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:38:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:38:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:38:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:38:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:29,255 [237] [DEBUG] [app] Ending request: urn:request:a5195649-4d73-4900-adc8-f47049f18af1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5195649-4d73-4900-adc8-f47049f18af1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:38:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:38:29,255 [238] [DEBUG] [app] Ending request: urn:request:56a53552-df60-4db6-b3b1-bca131a91e5c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:56a53552-df60-4db6-b3b1-bca131a91e5c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:38:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:38:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:38:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
repositoryactioncounter stdout | 2025-02-07 04:38:30,779 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:38:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:38:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:32.625945+00:00 (in 1.004341 seconds)
namespacegcworker stdout | 2025-02-07 04:38:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:31 UTC)" (scheduled at 2025-02-07 04:38:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:38:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:38:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 31, 621920), True, datetime.datetime(2025, 2, 7, 4, 38, 31, 621920), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:38:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:38:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:38:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:38:31,923 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:38:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:38:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:03.419110+00:00 (in 30.997933 seconds)
repositorygcworker stdout | 2025-02-07 04:38:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:43:32 UTC)" (scheduled at 2025-02-07 04:38:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:38:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 38, 32, 421475), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:38:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 38, 32, 421475), True, datetime.datetime(2025, 2, 7, 4, 38, 32, 421475), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:38:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 38, 32, 421475), True, datetime.datetime(2025, 2, 7, 4, 38, 32, 421475), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 38, 32, 421475), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:38:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:38:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:43:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:38:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:38:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:31.621131+00:00 (in 58.994702 seconds)
namespacegcworker stdout | 2025-02-07 04:38:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:32 UTC)" (scheduled at 2025-02-07 04:38:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:38:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:38:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:38:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:38:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:03.919286+00:00 (in 29.999579 seconds)
buildlogsarchiver stdout | 2025-02-07 04:38:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:03 UTC)" (scheduled at 2025-02-07 04:38:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:38:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 38, 33, 919951), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:38:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:38:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:38:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:38:36,274 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:38:37,313 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:56.713547+00:00 (in 18.997131 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:37 UTC)" (scheduled at 2025-02-07 04:38:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:38:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:38:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:48.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:38:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:48 UTC)" (scheduled at 2025-02-07 04:38:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:38:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:38:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 38, 732872), True, datetime.datetime(2025, 2, 7, 4, 38, 38, 732872), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:38:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:38:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:38:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:38:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:38:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:54.423055+00:00 (in 13.998359 seconds)
gcworker stdout | 2025-02-07 04:38:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:10 UTC)" (scheduled at 2025-02-07 04:38:40.424250+00:00)
gcworker stdout | 2025-02-07 04:38:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:38:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902820431, None, 1, 0])
gcworker stdout | 2025-02-07 04:38:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:38:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:38:41,234 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:38:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:38:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:11.831521+00:00 (in 29.999547 seconds)
autopruneworker stdout | 2025-02-07 04:38:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:11 UTC)" (scheduled at 2025-02-07 04:38:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:38:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899521836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:38:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:38:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:38:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:38:41,931 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:38:42,383 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:38:43,696 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:38:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:38:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:59.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 04:38:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:59 UTC)" (scheduled at 2025-02-07 04:38:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:38:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:38:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:38:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:38:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:38:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:38:44,235 [235] [DEBUG] [app] Starting request: urn:request:0c17d2a8-3601-4949-aff5-451b893b92b2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:44,235 [238] [DEBUG] [app] Starting request: urn:request:fb133ba9-a41b-4f10-b1c8-80baec4ef8d0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:38:44,238 [239] [DEBUG] [app] Starting request: urn:request:59219368-99b8-465d-ba0a-c39a9ab281f1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:44,238 [246] [DEBUG] [app] Starting request: urn:request:532c8b44-5f9a-4e8c-9e15-ff719afa95f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:44,238 [246] [DEBUG] [app] Ending request: urn:request:532c8b44-5f9a-4e8c-9e15-ff719afa95f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:532c8b44-5f9a-4e8c-9e15-ff719afa95f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:38:44,238 [239] [DEBUG] [app] Ending request: urn:request:59219368-99b8-465d-ba0a-c39a9ab281f1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59219368-99b8-465d-ba0a-c39a9ab281f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:38:44,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:38:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:38:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:44,241 [238] [DEBUG] [app] Starting request: urn:request:d71a6435-9988-4a36-a83a-604611562862 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:44,241 [238] [DEBUG] [app] Ending request: urn:request:d71a6435-9988-4a36-a83a-604611562862 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d71a6435-9988-4a36-a83a-604611562862', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:38:44,241 [237] [DEBUG] [app] Starting request: urn:request:6758405c-52c0-4ed4-9e80-de2b75c8bcaa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:44,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:38:44,242 [237] [DEBUG] [app] Ending request: urn:request:6758405c-52c0-4ed4-9e80-de2b75c8bcaa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6758405c-52c0-4ed4-9e80-de2b75c8bcaa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:38:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:38:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:38:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:38:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:38:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:44,253 [235] [DEBUG] [app] Ending request: urn:request:0c17d2a8-3601-4949-aff5-451b893b92b2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c17d2a8-3601-4949-aff5-451b893b92b2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:38:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:38:44,253 [238] [DEBUG] [app] Ending request: urn:request:fb133ba9-a41b-4f10-b1c8-80baec4ef8d0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb133ba9-a41b-4f10-b1c8-80baec4ef8d0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:38:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:38:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:38:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
notificationworker stdout | 2025-02-07 04:38:44,646 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:38:44,706 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:38:45,706 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:46,248 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:38:46,743 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:38:46,752 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:38:46,864 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:38:46,914 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:38:46,919 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:38:46,922 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:38:46,925 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:38:46,928 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:38:47,342 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:38:47,818 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:38:47,939 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:38:47,942 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:38:47,945 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:38:47,991 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:38:48,012 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:38:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:38:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:50.814677+00:00 (in 2.082097 seconds)
notificationworker stdout | 2025-02-07 04:38:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:58 UTC)" (scheduled at 2025-02-07 04:38:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:38:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:38:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 48, 732857), True, datetime.datetime(2025, 2, 7, 4, 38, 48, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:38:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:38:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:38:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:38:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:38:49,148 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:38:49,151 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:38:49,157 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:38:49,165 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:38:49,168 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:38:49,170 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:38:49,173 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:38:49,177 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:38:49,181 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:38:49,569 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:38:49,643 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:38:50,026 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:38:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:38:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:38:58.732147+00:00 (in 7.917001 seconds)
notificationworker stdout | 2025-02-07 04:38:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:50 UTC)" (scheduled at 2025-02-07 04:38:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:38:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:38:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:38:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:38:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:10.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 04:38:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:24 UTC)" (scheduled at 2025-02-07 04:38:54.423055+00:00)
gcworker stdout | 2025-02-07 04:38:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:38:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:38:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:38:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:56.118855+00:00 (in 59.999444 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:38:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:56 UTC)" (scheduled at 2025-02-07 04:38:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:38:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:38:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:38:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:37.715878+00:00 (in 41.001822 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:56 UTC)" (scheduled at 2025-02-07 04:38:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:56,715 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:38:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:38:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:38:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:08.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 04:38:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:08 UTC)" (scheduled at 2025-02-07 04:38:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:38:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:38:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 38, 58, 732887), True, datetime.datetime(2025, 2, 7, 4, 38, 58, 732887), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:38:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:38:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:38:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:38:59,048 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:38:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:38:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:14.225378+00:00 (in 14.999535 seconds)
quotatotalworker stdout | 2025-02-07 04:38:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:14 UTC)" (scheduled at 2025-02-07 04:38:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:38:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:38:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:38:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:38:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:38:59,235 [235] [DEBUG] [app] Starting request: urn:request:77838400-98b4-4404-a983-c67a34d3582a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:59,235 [238] [DEBUG] [app] Starting request: urn:request:48731ddb-c508-46a3-9d4f-8c60a008d3fc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:38:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:38:59,238 [239] [DEBUG] [app] Starting request: urn:request:c8bcfbad-a053-4e67-8f4c-cc0023509777 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:59,238 [245] [DEBUG] [app] Starting request: urn:request:e797c037-b78e-42ba-80af-db4e28ae7621 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:38:59,239 [239] [DEBUG] [app] Ending request: urn:request:c8bcfbad-a053-4e67-8f4c-cc0023509777 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c8bcfbad-a053-4e67-8f4c-cc0023509777', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:38:59,239 [245] [DEBUG] [app] Ending request: urn:request:e797c037-b78e-42ba-80af-db4e28ae7621 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e797c037-b78e-42ba-80af-db4e28ae7621', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:38:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:38:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:38:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:38:59,242 [238] [DEBUG] [app] Starting request: urn:request:45877589-5531-4b99-9943-0ad9236481e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:59,242 [238] [DEBUG] [app] Ending request: urn:request:45877589-5531-4b99-9943-0ad9236481e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45877589-5531-4b99-9943-0ad9236481e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:38:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:38:59,243 [237] [DEBUG] [app] Starting request: urn:request:71c09978-521a-4827-bc5b-c89954f3255d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:38:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:59,243 [237] [DEBUG] [app] Ending request: urn:request:71c09978-521a-4827-bc5b-c89954f3255d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71c09978-521a-4827-bc5b-c89954f3255d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:38:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:38:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:38:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:38:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:38:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:38:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:38:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:38:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:38:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:38:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:38:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:59,254 [238] [DEBUG] [app] Ending request: urn:request:48731ddb-c508-46a3-9d4f-8c60a008d3fc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:48731ddb-c508-46a3-9d4f-8c60a008d3fc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:38:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:38:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:38:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:38:59,256 [235] [DEBUG] [app] Ending request: urn:request:77838400-98b4-4404-a983-c67a34d3582a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77838400-98b4-4404-a983-c67a34d3582a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:38:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:38:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 04:38:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:38:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:38:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:38:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:04.936688+00:00 (in 4.998311 seconds)
exportactionlogsworker stdout | 2025-02-07 04:38:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:59 UTC)" (scheduled at 2025-02-07 04:38:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:38:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:38:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:39:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:39:00,794 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:39:01,959 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:39:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:39:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:23.420886+00:00 (in 20.001310 seconds)
repositorygcworker stdout | 2025-02-07 04:39:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:03 UTC)" (scheduled at 2025-02-07 04:39:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:39:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:39:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 3, 419861), True, datetime.datetime(2025, 2, 7, 4, 39, 3, 419861), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:39:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:39:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:39:03,426 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:39:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:39:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:33.919286+00:00 (in 29.999571 seconds)
buildlogsarchiver stdout | 2025-02-07 04:39:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:33 UTC)" (scheduled at 2025-02-07 04:39:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:39:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 39, 3, 919958), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:39:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:39:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:39:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:39:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:39:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:52.937648+00:00 (in 48.000545 seconds)
exportactionlogsworker stdout | 2025-02-07 04:39:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:04 UTC)" (scheduled at 2025-02-07 04:39:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:39:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:39:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 4, 937404), True, datetime.datetime(2025, 2, 7, 4, 39, 4, 937404), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:39:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:39:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:39:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:39:06,296 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:39:07,350 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:39:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:39:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:18.732147+00:00 (in 9.999533 seconds)
notificationworker stdout | 2025-02-07 04:39:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:18 UTC)" (scheduled at 2025-02-07 04:39:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:39:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:39:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 8, 732898), True, datetime.datetime(2025, 2, 7, 4, 39, 8, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:39:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:39:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:39:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:39:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:39:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:24.423055+00:00 (in 13.998388 seconds)
gcworker stdout | 2025-02-07 04:39:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:40 UTC)" (scheduled at 2025-02-07 04:39:10.424250+00:00)
gcworker stdout | 2025-02-07 04:39:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:39:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902850430, None, 1, 0])
gcworker stdout | 2025-02-07 04:39:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:39:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:39:11,270 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:39:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:39:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:41.831521+00:00 (in 29.999573 seconds)
autopruneworker stdout | 2025-02-07 04:39:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:41 UTC)" (scheduled at 2025-02-07 04:39:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:39:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899551836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:39:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:39:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:39:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:39:11,940 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:39:12,394 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:39:13,732 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:39:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:39:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:29.225378+00:00 (in 14.999543 seconds)
quotatotalworker stdout | 2025-02-07 04:39:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:29 UTC)" (scheduled at 2025-02-07 04:39:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:39:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:39:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:39:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:39:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:39:14,235 [235] [DEBUG] [app] Starting request: urn:request:d01d0016-4ad3-45ba-9afb-1ddbe69c4df4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:14,235 [237] [DEBUG] [app] Starting request: urn:request:11cf42a3-0d34-430c-bdca-fe0cbdc0c165 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:39:14,238 [246] [DEBUG] [app] Starting request: urn:request:a15d0546-db4e-4033-af90-27fa355a35ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:14,238 [239] [DEBUG] [app] Starting request: urn:request:5d725efd-3009-42a3-b519-af2998b70ce9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:14,238 [246] [DEBUG] [app] Ending request: urn:request:a15d0546-db4e-4033-af90-27fa355a35ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a15d0546-db4e-4033-af90-27fa355a35ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:39:14,238 [239] [DEBUG] [app] Ending request: urn:request:5d725efd-3009-42a3-b519-af2998b70ce9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d725efd-3009-42a3-b519-af2998b70ce9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:39:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:39:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:14,241 [237] [DEBUG] [app] Starting request: urn:request:de78b943-3492-4314-9019-5dc343c9dd1a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:39:14,242 [237] [DEBUG] [app] Ending request: urn:request:de78b943-3492-4314-9019-5dc343c9dd1a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de78b943-3492-4314-9019-5dc343c9dd1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:39:14,242 [238] [DEBUG] [app] Starting request: urn:request:1498960e-f6ce-4f2f-8edf-58037818f920 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:39:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:14,242 [238] [DEBUG] [app] Ending request: urn:request:1498960e-f6ce-4f2f-8edf-58037818f920 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1498960e-f6ce-4f2f-8edf-58037818f920', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:39:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:39:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:39:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:14,254 [237] [DEBUG] [app] Ending request: urn:request:11cf42a3-0d34-430c-bdca-fe0cbdc0c165 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11cf42a3-0d34-430c-bdca-fe0cbdc0c165', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:39:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:39:14,254 [235] [DEBUG] [app] Ending request: urn:request:d01d0016-4ad3-45ba-9afb-1ddbe69c4df4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d01d0016-4ad3-45ba-9afb-1ddbe69c4df4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:39:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
notificationworker stdout | 2025-02-07 04:39:14,673 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:39:14,714 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:39:15,743 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:16,274 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:39:16,759 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:39:16,773 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:39:16,887 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:39:16,925 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:39:16,929 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:39:16,932 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:39:16,935 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:39:16,939 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:39:17,370 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:39:17,842 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:39:17,948 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:39:17,951 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:39:17,954 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:39:18,010 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:39:18,032 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:39:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:39:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:28.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 04:39:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:28 UTC)" (scheduled at 2025-02-07 04:39:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:39:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:39:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 18, 732851), True, datetime.datetime(2025, 2, 7, 4, 39, 18, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:39:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:39:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:39:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:39:19,160 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:39:19,162 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:39:19,165 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:39:19,173 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:39:19,177 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:39:19,179 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:39:19,182 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:39:19,184 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:39:19,188 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:39:19,590 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:39:19,670 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:39:20,062 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:39:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:39:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:03.419110+00:00 (in 39.997715 seconds)
repositorygcworker stdout | 2025-02-07 04:39:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:23 UTC)" (scheduled at 2025-02-07 04:39:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:39:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:39:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:39:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:39:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:40.424250+00:00 (in 16.000693 seconds)
gcworker stdout | 2025-02-07 04:39:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:54 UTC)" (scheduled at 2025-02-07 04:39:24.423055+00:00)
gcworker stdout | 2025-02-07 04:39:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:39:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:39:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:39:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:39:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:38.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 04:39:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:38 UTC)" (scheduled at 2025-02-07 04:39:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:39:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:39:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 28, 732855), True, datetime.datetime(2025, 2, 7, 4, 39, 28, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:39:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:39:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:39:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:39:29,084 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:39:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:39:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:44.225378+00:00 (in 14.999534 seconds)
quotatotalworker stdout | 2025-02-07 04:39:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:44 UTC)" (scheduled at 2025-02-07 04:39:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:39:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:39:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:39:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:39:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:39:29,235 [235] [DEBUG] [app] Starting request: urn:request:86062ed3-c3e7-4b8b-a5bb-8a0a8233e04a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:29,235 [237] [DEBUG] [app] Starting request: urn:request:8af856f1-2614-439e-8145-dbe964a6e08b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:39:29,238 [246] [DEBUG] [app] Starting request: urn:request:07247562-198b-4510-bb57-55d5b0da3c1b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:29,239 [246] [DEBUG] [app] Ending request: urn:request:07247562-198b-4510-bb57-55d5b0da3c1b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07247562-198b-4510-bb57-55d5b0da3c1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:39:29,239 [239] [DEBUG] [app] Starting request: urn:request:601f7106-61c5-4d2a-a6b0-c836aa9f384c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:39:29,239 [239] [DEBUG] [app] Ending request: urn:request:601f7106-61c5-4d2a-a6b0-c836aa9f384c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:601f7106-61c5-4d2a-a6b0-c836aa9f384c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:39:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:39:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:29,242 [237] [DEBUG] [app] Starting request: urn:request:751d1907-1297-4fff-8617-f2b00325cd7e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:39:29,242 [237] [DEBUG] [app] Ending request: urn:request:751d1907-1297-4fff-8617-f2b00325cd7e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:751d1907-1297-4fff-8617-f2b00325cd7e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:39:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:29,243 [238] [DEBUG] [app] Starting request: urn:request:537d9758-f872-4a58-88f0-ca3f9e6ffe61 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:39:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:29,243 [238] [DEBUG] [app] Ending request: urn:request:537d9758-f872-4a58-88f0-ca3f9e6ffe61 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:537d9758-f872-4a58-88f0-ca3f9e6ffe61', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:39:29,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:39:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:39:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:29,255 [237] [DEBUG] [app] Ending request: urn:request:8af856f1-2614-439e-8145-dbe964a6e08b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8af856f1-2614-439e-8145-dbe964a6e08b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:39:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:39:29,255 [235] [DEBUG] [app] Ending request: urn:request:86062ed3-c3e7-4b8b-a5bb-8a0a8233e04a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:86062ed3-c3e7-4b8b-a5bb-8a0a8233e04a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:39:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
repositoryactioncounter stdout | 2025-02-07 04:39:30,830 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:39:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:39:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:32.625945+00:00 (in 1.004355 seconds)
namespacegcworker stdout | 2025-02-07 04:39:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:31 UTC)" (scheduled at 2025-02-07 04:39:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:39:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:39:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 31, 621863), True, datetime.datetime(2025, 2, 7, 4, 39, 31, 621863), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:39:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:39:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:39:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:39:31,986 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:39:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:39:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:31.621131+00:00 (in 58.994721 seconds)
namespacegcworker stdout | 2025-02-07 04:39:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:32 UTC)" (scheduled at 2025-02-07 04:39:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:39:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:39:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:39:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:39:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:03.919286+00:00 (in 29.999569 seconds)
buildlogsarchiver stdout | 2025-02-07 04:39:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:03 UTC)" (scheduled at 2025-02-07 04:39:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:39:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 39, 33, 919967), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:39:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:39:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:39:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:39:36,314 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:39:37,387 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:56.713547+00:00 (in 18.997214 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:37 UTC)" (scheduled at 2025-02-07 04:39:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:39:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:39:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:48.732147+00:00 (in 9.999578 seconds)
notificationworker stdout | 2025-02-07 04:39:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:48 UTC)" (scheduled at 2025-02-07 04:39:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:39:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:39:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 38, 732872), True, datetime.datetime(2025, 2, 7, 4, 39, 38, 732872), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:39:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:39:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:39:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:39:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:39:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:54.423055+00:00 (in 13.998302 seconds)
gcworker stdout | 2025-02-07 04:39:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:10 UTC)" (scheduled at 2025-02-07 04:39:40.424250+00:00)
gcworker stdout | 2025-02-07 04:39:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:39:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902880430, None, 1, 0])
gcworker stdout | 2025-02-07 04:39:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:39:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:39:41,306 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:39:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:39:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:11.831521+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2025-02-07 04:39:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:11 UTC)" (scheduled at 2025-02-07 04:39:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:39:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899581836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:39:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:39:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:39:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:39:41,973 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:39:42,407 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:39:43,744 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:39:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:39:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:59.225378+00:00 (in 14.999533 seconds)
quotatotalworker stdout | 2025-02-07 04:39:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:59 UTC)" (scheduled at 2025-02-07 04:39:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:39:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:39:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:39:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:39:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:39:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:39:44,235 [237] [DEBUG] [app] Starting request: urn:request:2dd4d37c-f6bb-4a5d-af71-1126ef60409a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:44,235 [235] [DEBUG] [app] Starting request: urn:request:cfb9446c-8ff8-4e43-8e5c-6ab6268016af (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:39:44,238 [245] [DEBUG] [app] Starting request: urn:request:3a979fb6-8031-4bb0-b8ef-11664fdbd00f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:44,238 [246] [DEBUG] [app] Starting request: urn:request:6d8537b5-e9d2-45fc-8c9e-16bf9bdb7bbe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:44,238 [245] [DEBUG] [app] Ending request: urn:request:3a979fb6-8031-4bb0-b8ef-11664fdbd00f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a979fb6-8031-4bb0-b8ef-11664fdbd00f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:39:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:39:44,239 [246] [DEBUG] [app] Ending request: urn:request:6d8537b5-e9d2-45fc-8c9e-16bf9bdb7bbe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d8537b5-e9d2-45fc-8c9e-16bf9bdb7bbe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:39:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:39:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:44,241 [235] [DEBUG] [app] Starting request: urn:request:16058c57-aded-4985-b55f-a8cdea61facc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:39:44,241 [235] [DEBUG] [app] Ending request: urn:request:16058c57-aded-4985-b55f-a8cdea61facc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16058c57-aded-4985-b55f-a8cdea61facc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:39:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:44,242 [238] [DEBUG] [app] Starting request: urn:request:08ec5095-b003-48c1-8a10-48fde7838206 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:39:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:44,242 [238] [DEBUG] [app] Ending request: urn:request:08ec5095-b003-48c1-8a10-48fde7838206 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:08ec5095-b003-48c1-8a10-48fde7838206', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:39:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:39:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:44,253 [235] [DEBUG] [app] Ending request: urn:request:cfb9446c-8ff8-4e43-8e5c-6ab6268016af (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cfb9446c-8ff8-4e43-8e5c-6ab6268016af', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:39:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:39:44,254 [237] [DEBUG] [app] Ending request: urn:request:2dd4d37c-f6bb-4a5d-af71-1126ef60409a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2dd4d37c-f6bb-4a5d-af71-1126ef60409a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:39:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:39:44,710 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:39:44,728 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:39:45,774 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:46,310 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:39:46,794 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:39:46,797 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:39:46,924 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:39:46,931 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:39:46,936 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:39:46,939 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:39:46,942 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:39:46,946 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:39:47,378 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:39:47,878 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:39:47,955 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:39:47,958 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:39:47,961 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:39:48,017 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:39:48,068 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:39:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:39:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:50.814677+00:00 (in 2.082105 seconds)
notificationworker stdout | 2025-02-07 04:39:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:58 UTC)" (scheduled at 2025-02-07 04:39:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:39:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:39:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 48, 732854), True, datetime.datetime(2025, 2, 7, 4, 39, 48, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:39:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:39:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:39:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:39:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:39:49,170 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:39:49,173 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:39:49,177 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:39:49,181 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:39:49,183 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:39:49,186 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:39:49,189 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:39:49,192 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:39:49,196 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
namespacegcworker stdout | 2025-02-07 04:39:49,626 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:39:49,689 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:39:50,098 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:39:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:39:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:58.732147+00:00 (in 7.916973 seconds)
notificationworker stdout | 2025-02-07 04:39:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:50 UTC)" (scheduled at 2025-02-07 04:39:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:39:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:39:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:39:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:39:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:39:59.937899+00:00 (in 6.999794 seconds)
exportactionlogsworker stdout | 2025-02-07 04:39:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:44:52 UTC)" (scheduled at 2025-02-07 04:39:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:39:52,939 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 39, 52, 938392), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:39:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 39, 52, 938392), True, datetime.datetime(2025, 2, 7, 4, 39, 52, 938392), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:39:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 39, 52, 938392), True, datetime.datetime(2025, 2, 7, 4, 39, 52, 938392), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 39, 52, 938392), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:39:52,949 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:39:52,949 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:44:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:39:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:39:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:10.424250+00:00 (in 16.000718 seconds)
gcworker stdout | 2025-02-07 04:39:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:24 UTC)" (scheduled at 2025-02-07 04:39:54.423055+00:00)
gcworker stdout | 2025-02-07 04:39:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:39:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:39:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:39:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:56.118855+00:00 (in 59.999496 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:39:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:56 UTC)" (scheduled at 2025-02-07 04:39:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:39:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:39:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:39:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:37.715878+00:00 (in 41.001867 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:56 UTC)" (scheduled at 2025-02-07 04:39:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:39:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:39:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:39:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:08.732147+00:00 (in 9.999500 seconds)
notificationworker stdout | 2025-02-07 04:39:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:08 UTC)" (scheduled at 2025-02-07 04:39:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:39:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:39:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 39, 58, 732914), True, datetime.datetime(2025, 2, 7, 4, 39, 58, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:39:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:39:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:39:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:39:59,120 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:39:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:39:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:14.225378+00:00 (in 14.999543 seconds)
quotatotalworker stdout | 2025-02-07 04:39:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:14 UTC)" (scheduled at 2025-02-07 04:39:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:39:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:39:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:39:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:39:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:39:59,236 [237] [DEBUG] [app] Starting request: urn:request:70fcdf46-bdd1-471f-b7c8-db798522ff63 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:59,236 [235] [DEBUG] [app] Starting request: urn:request:a39ee7f7-ffed-44d7-8504-15ff498bc111 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:39:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:39:59,239 [246] [DEBUG] [app] Starting request: urn:request:b0071683-ad92-4c11-b5ec-65620c769fe5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:59,239 [239] [DEBUG] [app] Starting request: urn:request:41acb002-fb82-4690-9e96-67a9ee2affb7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:39:59,239 [246] [DEBUG] [app] Ending request: urn:request:b0071683-ad92-4c11-b5ec-65620c769fe5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b0071683-ad92-4c11-b5ec-65620c769fe5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:39:59,239 [239] [DEBUG] [app] Ending request: urn:request:41acb002-fb82-4690-9e96-67a9ee2affb7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:41acb002-fb82-4690-9e96-67a9ee2affb7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:39:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:39:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:39:59,242 [237] [DEBUG] [app] Starting request: urn:request:b3cde38c-3d7f-4fce-9e7e-2a31e370d841 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:39:59,242 [237] [DEBUG] [app] Ending request: urn:request:b3cde38c-3d7f-4fce-9e7e-2a31e370d841 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3cde38c-3d7f-4fce-9e7e-2a31e370d841', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:59,242 [236] [DEBUG] [app] Starting request: urn:request:fad64e71-c8ab-408d-8a62-000a023e1096 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:39:59,242 [236] [DEBUG] [app] Ending request: urn:request:fad64e71-c8ab-408d-8a62-000a023e1096 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fad64e71-c8ab-408d-8a62-000a023e1096', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:39:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:39:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:39:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:39:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:39:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:39:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:39:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:39:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:39:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:39:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:39:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:39:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:39:59,253 [237] [DEBUG] [app] Ending request: urn:request:70fcdf46-bdd1-471f-b7c8-db798522ff63 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:70fcdf46-bdd1-471f-b7c8-db798522ff63', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:39:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:39:59,254 [235] [DEBUG] [app] Ending request: urn:request:a39ee7f7-ffed-44d7-8504-15ff498bc111 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a39ee7f7-ffed-44d7-8504-15ff498bc111', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:39:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:39:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:39:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:39:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:39:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:39:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:04.936688+00:00 (in 4.998322 seconds)
exportactionlogsworker stdout | 2025-02-07 04:39:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:59 UTC)" (scheduled at 2025-02-07 04:39:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:39:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:39:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:40:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:40:00,867 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:40:02,022 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:40:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:40:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:23.420886+00:00 (in 20.001293 seconds)
repositorygcworker stdout | 2025-02-07 04:40:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:03 UTC)" (scheduled at 2025-02-07 04:40:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:40:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:40:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 3, 419879), True, datetime.datetime(2025, 2, 7, 4, 40, 3, 419879), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:40:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:40:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:40:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:40:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:40:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:33.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 04:40:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:33 UTC)" (scheduled at 2025-02-07 04:40:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:40:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 40, 3, 920006), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:40:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:40:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:40:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:40:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:40:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:59.937899+00:00 (in 55.000778 seconds)
exportactionlogsworker stdout | 2025-02-07 04:40:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:04 UTC)" (scheduled at 2025-02-07 04:40:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:40:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:40:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 4, 937408), True, datetime.datetime(2025, 2, 7, 4, 40, 4, 937408), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:40:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:40:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:40:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:40:06,350 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:40:07,407 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:40:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:40:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:18.732147+00:00 (in 9.999499 seconds)
notificationworker stdout | 2025-02-07 04:40:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:18 UTC)" (scheduled at 2025-02-07 04:40:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:40:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:40:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 8, 732968), True, datetime.datetime(2025, 2, 7, 4, 40, 8, 732968), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:40:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:40:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:40:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:40:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:40:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:24.423055+00:00 (in 13.998362 seconds)
gcworker stdout | 2025-02-07 04:40:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:40 UTC)" (scheduled at 2025-02-07 04:40:10.424250+00:00)
gcworker stdout | 2025-02-07 04:40:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:40:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902910431, None, 1, 0])
gcworker stdout | 2025-02-07 04:40:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:40:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:40:11,342 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:40:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:40:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:41.831521+00:00 (in 29.999534 seconds)
autopruneworker stdout | 2025-02-07 04:40:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:41 UTC)" (scheduled at 2025-02-07 04:40:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:40:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899611836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:40:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:40:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:40:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:40:12,003 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:40:12,435 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:40:13,774 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:40:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:40:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:29.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 04:40:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:29 UTC)" (scheduled at 2025-02-07 04:40:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:40:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:40:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:40:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:40:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:40:14,235 [238] [DEBUG] [app] Starting request: urn:request:f9c15f98-663d-4708-b246-e8843342dd95 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:14,235 [235] [DEBUG] [app] Starting request: urn:request:b18e319b-da49-4834-87c9-d071c0adf21a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:40:14,238 [239] [DEBUG] [app] Starting request: urn:request:2ba4bc11-fed9-4b3c-a395-ccedbed0a413 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:14,239 [245] [DEBUG] [app] Starting request: urn:request:924278f4-960b-452e-82f3-653ae1e8f15b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:14,239 [239] [DEBUG] [app] Ending request: urn:request:2ba4bc11-fed9-4b3c-a395-ccedbed0a413 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ba4bc11-fed9-4b3c-a395-ccedbed0a413', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:40:14,239 [245] [DEBUG] [app] Ending request: urn:request:924278f4-960b-452e-82f3-653ae1e8f15b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:924278f4-960b-452e-82f3-653ae1e8f15b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:40:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:40:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:14,242 [235] [DEBUG] [app] Starting request: urn:request:6a5cb843-c511-41e0-823b-1c610e6846a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:14,242 [235] [DEBUG] [app] Ending request: urn:request:6a5cb843-c511-41e0-823b-1c610e6846a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a5cb843-c511-41e0-823b-1c610e6846a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:40:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:14,243 [237] [DEBUG] [app] Starting request: urn:request:c415b643-5949-4ffe-8802-a9866ef12e63 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:14,243 [237] [DEBUG] [app] Ending request: urn:request:c415b643-5949-4ffe-8802-a9866ef12e63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c415b643-5949-4ffe-8802-a9866ef12e63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:40:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:14,254 [235] [DEBUG] [app] Ending request: urn:request:b18e319b-da49-4834-87c9-d071c0adf21a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b18e319b-da49-4834-87c9-d071c0adf21a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:40:14,254 [238] [DEBUG] [app] Ending request: urn:request:f9c15f98-663d-4708-b246-e8843342dd95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f9c15f98-663d-4708-b246-e8843342dd95', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:40:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:40:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:40:14,734 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:40:14,764 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:40:15,802 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:16,332 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:40:16,822 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:40:16,829 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:40:16,937 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-web stdout | 2025-02-07 04:40:16,940 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:40:16,942 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:40:16,945 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:40:16,949 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:40:16,952 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
repositorygcworker stdout | 2025-02-07 04:40:17,401 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:40:17,914 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:40:17,963 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:40:17,967 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:40:17,972 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:40:18,046 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:40:18,104 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:40:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:40:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:28.732147+00:00 (in 9.999514 seconds)
notificationworker stdout | 2025-02-07 04:40:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:28 UTC)" (scheduled at 2025-02-07 04:40:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:40:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:40:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 18, 732916), True, datetime.datetime(2025, 2, 7, 4, 40, 18, 732916), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:40:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:40:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:40:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:40:19,179 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:40:19,182 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:40:19,187 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:40:19,189 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:40:19,192 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:40:19,195 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:40:19,198 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:40:19,203 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:40:19,205 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:40:19,662 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:40:19,717 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:40:20,128 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:40:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:40:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:03.419110+00:00 (in 39.997774 seconds)
repositorygcworker stdout | 2025-02-07 04:40:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:23 UTC)" (scheduled at 2025-02-07 04:40:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:40:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:40:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:40:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:40:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:40.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 04:40:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:54 UTC)" (scheduled at 2025-02-07 04:40:24.423055+00:00)
gcworker stdout | 2025-02-07 04:40:24,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 04:40:24,430 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:40:24,430 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:40:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:40:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:40:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:40:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:38.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 04:40:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:38 UTC)" (scheduled at 2025-02-07 04:40:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:40:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:40:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 28, 732844), True, datetime.datetime(2025, 2, 7, 4, 40, 28, 732844), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:40:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:40:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:40:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:40:29,141 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:40:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:40:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:44.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 04:40:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:44 UTC)" (scheduled at 2025-02-07 04:40:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:40:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:40:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:40:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:40:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:40:29,235 [235] [DEBUG] [app] Starting request: urn:request:11d03d9c-abd7-41c0-9b7a-32f3dbb7aefa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:29,235 [238] [DEBUG] [app] Starting request: urn:request:9b0cb423-96ed-4b5c-8e90-44d53f496b14 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:40:29,238 [246] [DEBUG] [app] Starting request: urn:request:91f4132e-f8d9-44c8-9260-d097eddaadaf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:29,238 [239] [DEBUG] [app] Starting request: urn:request:16a83323-0caa-46b7-8adc-6b2fd41201ed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:29,238 [246] [DEBUG] [app] Ending request: urn:request:91f4132e-f8d9-44c8-9260-d097eddaadaf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:91f4132e-f8d9-44c8-9260-d097eddaadaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:40:29,238 [239] [DEBUG] [app] Ending request: urn:request:16a83323-0caa-46b7-8adc-6b2fd41201ed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:16a83323-0caa-46b7-8adc-6b2fd41201ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:40:29,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:40:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:29,238 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:29,238 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:29,239 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:29,240 [238] [DEBUG] [app] Starting request: urn:request:a71ea667-9923-4668-b560-34de5af03309 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:29,241 [238] [DEBUG] [app] Ending request: urn:request:a71ea667-9923-4668-b560-34de5af03309 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a71ea667-9923-4668-b560-34de5af03309', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:40:29,241 [236] [DEBUG] [app] Starting request: urn:request:4a8bde0c-ac7b-4584-bda1-351d518e9522 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:29,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:40:29,241 [236] [DEBUG] [app] Ending request: urn:request:4a8bde0c-ac7b-4584-bda1-351d518e9522 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a8bde0c-ac7b-4584-bda1-351d518e9522', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:40:29,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:40:29,241 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:29,245 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:29,245 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:29,245 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:29,245 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:29,249 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:29,252 [238] [DEBUG] [app] Ending request: urn:request:9b0cb423-96ed-4b5c-8e90-44d53f496b14 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b0cb423-96ed-4b5c-8e90-44d53f496b14', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:29,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:40:29,252 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:40:29,252 [235] [DEBUG] [app] Ending request: urn:request:11d03d9c-abd7-41c0-9b7a-32f3dbb7aefa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11d03d9c-abd7-41c0-9b7a-32f3dbb7aefa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:40:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
repositoryactioncounter stdout | 2025-02-07 04:40:30,887 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:40:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:40:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:32.625945+00:00 (in 1.004333 seconds)
namespacegcworker stdout | 2025-02-07 04:40:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:31 UTC)" (scheduled at 2025-02-07 04:40:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:40:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:40:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 31, 621922), True, datetime.datetime(2025, 2, 7, 4, 40, 31, 621922), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:40:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:40:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:40:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:40:32,058 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:40:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:40:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:31.621131+00:00 (in 58.994661 seconds)
namespacegcworker stdout | 2025-02-07 04:40:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:32 UTC)" (scheduled at 2025-02-07 04:40:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:40:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:40:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:40:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:40:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:03.919286+00:00 (in 29.999563 seconds)
buildlogsarchiver stdout | 2025-02-07 04:40:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:03 UTC)" (scheduled at 2025-02-07 04:40:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:40:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 40, 33, 919973), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:40:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:40:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:40:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:40:36,386 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:40:37,443 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:56.713547+00:00 (in 18.997173 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:37 UTC)" (scheduled at 2025-02-07 04:40:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:40:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:40:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:48.732147+00:00 (in 9.999575 seconds)
notificationworker stdout | 2025-02-07 04:40:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:48 UTC)" (scheduled at 2025-02-07 04:40:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:40:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:40:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 38, 732873), True, datetime.datetime(2025, 2, 7, 4, 40, 38, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:40:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:40:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:40:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:40:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:40:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:54.423055+00:00 (in 13.998349 seconds)
gcworker stdout | 2025-02-07 04:40:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:10 UTC)" (scheduled at 2025-02-07 04:40:40.424250+00:00)
gcworker stdout | 2025-02-07 04:40:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:40:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902940431, None, 1, 0])
gcworker stdout | 2025-02-07 04:40:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:40:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:40:41,379 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:40:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:40:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:11.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 04:40:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:11 UTC)" (scheduled at 2025-02-07 04:40:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:40:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899641836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:40:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:40:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:40:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:40:42,018 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:40:42,472 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:40:43,792 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:40:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:40:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:59.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 04:40:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:59 UTC)" (scheduled at 2025-02-07 04:40:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:40:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:40:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:40:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:40:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:40:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:40:44,235 [238] [DEBUG] [app] Starting request: urn:request:47547d1d-25cb-4197-892f-18e3c7a34ece (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:44,235 [235] [DEBUG] [app] Starting request: urn:request:3ca4ac65-eece-4967-9fd8-77e2ad931214 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:40:44,239 [239] [DEBUG] [app] Starting request: urn:request:6107de66-8252-4513-866f-32670729afa9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:44,239 [246] [DEBUG] [app] Starting request: urn:request:37469c57-6dbc-4b12-8212-238f9c3c7359 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:44,239 [246] [DEBUG] [app] Ending request: urn:request:37469c57-6dbc-4b12-8212-238f9c3c7359 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:37469c57-6dbc-4b12-8212-238f9c3c7359', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:40:44,239 [239] [DEBUG] [app] Ending request: urn:request:6107de66-8252-4513-866f-32670729afa9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6107de66-8252-4513-866f-32670729afa9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:40:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:40:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:44,242 [235] [DEBUG] [app] Starting request: urn:request:2b29177e-1135-481f-833d-74bc7f09a600 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:44,242 [238] [DEBUG] [app] Starting request: urn:request:052cebd8-a037-41bc-808d-483af595a9c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:44,242 [238] [DEBUG] [app] Ending request: urn:request:052cebd8-a037-41bc-808d-483af595a9c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:052cebd8-a037-41bc-808d-483af595a9c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:40:44,242 [235] [DEBUG] [app] Ending request: urn:request:2b29177e-1135-481f-833d-74bc7f09a600 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2b29177e-1135-481f-833d-74bc7f09a600', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:40:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:44,254 [238] [DEBUG] [app] Ending request: urn:request:47547d1d-25cb-4197-892f-18e3c7a34ece (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47547d1d-25cb-4197-892f-18e3c7a34ece', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:44,254 [235] [DEBUG] [app] Ending request: urn:request:3ca4ac65-eece-4967-9fd8-77e2ad931214 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ca4ac65-eece-4967-9fd8-77e2ad931214', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:40:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:40:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:40:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:40:44,748 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:40:44,801 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:40:45,839 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:46,350 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:40:46,858 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:40:46,864 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-web stdout | 2025-02-07 04:40:46,947 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:40:46,951 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:40:46,954 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:40:46,957 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:40:46,959 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
exportactionlogsworker stdout | 2025-02-07 04:40:46,972 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:40:47,438 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:40:47,942 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:40:47,970 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:40:47,975 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:40:47,979 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:40:48,078 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:40:48,126 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:40:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:40:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:50.814677+00:00 (in 2.082100 seconds)
notificationworker stdout | 2025-02-07 04:40:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:58 UTC)" (scheduled at 2025-02-07 04:40:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:40:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:40:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 48, 732846), True, datetime.datetime(2025, 2, 7, 4, 40, 48, 732846), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:40:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:40:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:40:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:40:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:40:49,189 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:40:49,192 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:40:49,197 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:40:49,199 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:40:49,202 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:40:49,206 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:40:49,209 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:40:49,213 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:40:49,217 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:40:49,672 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:40:49,750 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:40:50,164 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:40:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:40:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:40:58.732147+00:00 (in 7.917014 seconds)
notificationworker stdout | 2025-02-07 04:40:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:50 UTC)" (scheduled at 2025-02-07 04:40:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:40:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:40:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:40:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:40:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:10.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 04:40:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:24 UTC)" (scheduled at 2025-02-07 04:40:54.423055+00:00)
gcworker stdout | 2025-02-07 04:40:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:40:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:40:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:40:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:56.118855+00:00 (in 59.999515 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:40:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:56 UTC)" (scheduled at 2025-02-07 04:40:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:40:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:40:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:40:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:37.715878+00:00 (in 41.001882 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:56 UTC)" (scheduled at 2025-02-07 04:40:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:40:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:40:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:40:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:08.732147+00:00 (in 9.999578 seconds)
notificationworker stdout | 2025-02-07 04:40:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:08 UTC)" (scheduled at 2025-02-07 04:40:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:40:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:40:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 40, 58, 732840), True, datetime.datetime(2025, 2, 7, 4, 40, 58, 732840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:40:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:40:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:40:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:40:59,177 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:40:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:40:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:14.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 04:40:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:14 UTC)" (scheduled at 2025-02-07 04:40:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:40:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:40:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:40:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:40:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:40:59,236 [235] [DEBUG] [app] Starting request: urn:request:1be1eee6-170d-4e95-bee9-c37bd1d2a16c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:59,236 [237] [DEBUG] [app] Starting request: urn:request:373ac97f-4170-489d-9563-491b36f42ac6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:40:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:40:59,239 [245] [DEBUG] [app] Starting request: urn:request:35923edf-e5ca-43ac-b595-a9de5ce52a6c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:59,239 [244] [DEBUG] [app] Starting request: urn:request:1870d15c-25e9-4ab4-9ef1-ea74e730cddc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:40:59,239 [245] [DEBUG] [app] Ending request: urn:request:35923edf-e5ca-43ac-b595-a9de5ce52a6c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35923edf-e5ca-43ac-b595-a9de5ce52a6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:40:59,239 [244] [DEBUG] [app] Ending request: urn:request:1870d15c-25e9-4ab4-9ef1-ea74e730cddc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1870d15c-25e9-4ab4-9ef1-ea74e730cddc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:40:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:40:59,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:59,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:40:59,242 [235] [DEBUG] [app] Starting request: urn:request:11e4f8b2-aa89-4643-95a2-d64c71e80336 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:59,242 [235] [DEBUG] [app] Ending request: urn:request:11e4f8b2-aa89-4643-95a2-d64c71e80336 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11e4f8b2-aa89-4643-95a2-d64c71e80336', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:40:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:59,244 [238] [DEBUG] [app] Starting request: urn:request:e6f9c98c-e9f3-4d34-be7e-4ac4cea88f30 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:40:59,244 [238] [DEBUG] [app] Ending request: urn:request:e6f9c98c-e9f3-4d34-be7e-4ac4cea88f30 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6f9c98c-e9f3-4d34-be7e-4ac4cea88f30', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:40:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:40:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:40:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:40:59,245 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:40:59,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:40:59,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:40:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:59,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:40:59,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:40:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:40:59,254 [235] [DEBUG] [app] Ending request: urn:request:1be1eee6-170d-4e95-bee9-c37bd1d2a16c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1be1eee6-170d-4e95-bee9-c37bd1d2a16c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:59,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:40:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:40:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:40:59,256 [237] [DEBUG] [app] Ending request: urn:request:373ac97f-4170-489d-9563-491b36f42ac6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:373ac97f-4170-489d-9563-491b36f42ac6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:40:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:40:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:40:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:40:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 04:40:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:40:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:04.936688+00:00 (in 4.998332 seconds)
exportactionlogsworker stdout | 2025-02-07 04:40:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:59 UTC)" (scheduled at 2025-02-07 04:40:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:40:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:40:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:41:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:41:00,923 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:41:02,095 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:41:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:41:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:23.420886+00:00 (in 20.001330 seconds)
repositorygcworker stdout | 2025-02-07 04:41:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:03 UTC)" (scheduled at 2025-02-07 04:41:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:41:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:41:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 3, 419834), True, datetime.datetime(2025, 2, 7, 4, 41, 3, 419834), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:41:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:41:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:41:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:41:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:41:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:33.919286+00:00 (in 29.999558 seconds)
buildlogsarchiver stdout | 2025-02-07 04:41:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:33 UTC)" (scheduled at 2025-02-07 04:41:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:41:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 41, 3, 919985), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:41:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:41:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:41:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:41:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:41:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:59.937899+00:00 (in 55.000773 seconds)
exportactionlogsworker stdout | 2025-02-07 04:41:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:04 UTC)" (scheduled at 2025-02-07 04:41:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:41:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:41:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 4, 937428), True, datetime.datetime(2025, 2, 7, 4, 41, 4, 937428), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:41:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:41:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:41:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:41:06,411 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:41:07,450 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:41:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:41:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:18.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 04:41:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:18 UTC)" (scheduled at 2025-02-07 04:41:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:41:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:41:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 8, 732851), True, datetime.datetime(2025, 2, 7, 4, 41, 8, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:41:08,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:41:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:41:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:41:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:41:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:24.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 04:41:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:40 UTC)" (scheduled at 2025-02-07 04:41:10.424250+00:00)
gcworker stdout | 2025-02-07 04:41:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:41:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738902970431, None, 1, 0])
gcworker stdout | 2025-02-07 04:41:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:41:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:41:11,400 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:41:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:41:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:41.831521+00:00 (in 29.999542 seconds)
autopruneworker stdout | 2025-02-07 04:41:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:41 UTC)" (scheduled at 2025-02-07 04:41:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:41:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899671836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:41:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:41:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:41:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:41:12,046 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:41:12,508 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:41:13,828 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:41:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:41:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:29.225378+00:00 (in 14.999517 seconds)
quotatotalworker stdout | 2025-02-07 04:41:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:29 UTC)" (scheduled at 2025-02-07 04:41:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:41:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:41:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:41:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:41:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:41:14,235 [238] [DEBUG] [app] Starting request: urn:request:14b4151c-683f-4374-93c7-af4c3e72f733 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:14,235 [235] [DEBUG] [app] Starting request: urn:request:9ba02896-4b52-4fb4-b177-cb1645bd5996 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:41:14,238 [246] [DEBUG] [app] Starting request: urn:request:4a049b45-bade-47ba-81bf-561acd9480b5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:14,238 [239] [DEBUG] [app] Starting request: urn:request:2af55309-0dbf-4549-a93e-7b99d80ef001 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:14,239 [246] [DEBUG] [app] Ending request: urn:request:4a049b45-bade-47ba-81bf-561acd9480b5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4a049b45-bade-47ba-81bf-561acd9480b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:41:14,239 [239] [DEBUG] [app] Ending request: urn:request:2af55309-0dbf-4549-a93e-7b99d80ef001 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2af55309-0dbf-4549-a93e-7b99d80ef001', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:41:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:41:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:41:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:14,242 [237] [DEBUG] [app] Starting request: urn:request:41e3ae7b-db1f-4c80-b383-96e0fd2cf8d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:14,242 [235] [DEBUG] [app] Starting request: urn:request:59b0aa4c-70bc-4197-b478-9006bc7ce888 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:14,242 [237] [DEBUG] [app] Ending request: urn:request:41e3ae7b-db1f-4c80-b383-96e0fd2cf8d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:41e3ae7b-db1f-4c80-b383-96e0fd2cf8d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:41:14,242 [235] [DEBUG] [app] Ending request: urn:request:59b0aa4c-70bc-4197-b478-9006bc7ce888 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59b0aa4c-70bc-4197-b478-9006bc7ce888', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:41:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:41:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:41:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:41:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:14,253 [238] [DEBUG] [app] Ending request: urn:request:14b4151c-683f-4374-93c7-af4c3e72f733 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14b4151c-683f-4374-93c7-af4c3e72f733', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:41:14,253 [235] [DEBUG] [app] Ending request: urn:request:9ba02896-4b52-4fb4-b177-cb1645bd5996 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ba02896-4b52-4fb4-b177-cb1645bd5996', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:41:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:41:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
notificationworker stdout | 2025-02-07 04:41:14,755 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:41:14,826 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:41:15,853 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:16,362 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:41:16,887 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:41:16,889 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-web stdout | 2025-02-07 04:41:16,954 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:41:16,958 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:41:16,960 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:41:16,964 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:41:16,966 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
exportactionlogsworker stdout | 2025-02-07 04:41:17,007 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:41:17,474 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:41:17,960 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:41:17,978 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:41:17,982 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:41:17,985 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:41:18,114 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:41:18,140 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:41:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:41:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:28.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 04:41:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:28 UTC)" (scheduled at 2025-02-07 04:41:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:41:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:41:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 18, 732866), True, datetime.datetime(2025, 2, 7, 4, 41, 18, 732866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:41:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:41:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:41:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:41:19,199 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:41:19,204 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:41:19,208 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:41:19,210 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:41:19,213 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:41:19,215 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:41:19,221 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:41:19,223 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:41:19,226 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:41:19,706 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:41:19,762 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:41:20,182 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:41:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:41:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:03.419110+00:00 (in 39.997782 seconds)
repositorygcworker stdout | 2025-02-07 04:41:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:23 UTC)" (scheduled at 2025-02-07 04:41:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:41:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:41:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:41:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:41:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:40.424250+00:00 (in 16.000763 seconds)
gcworker stdout | 2025-02-07 04:41:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:54 UTC)" (scheduled at 2025-02-07 04:41:24.423055+00:00)
gcworker stdout | 2025-02-07 04:41:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:41:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:41:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:41:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:41:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:38.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 04:41:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:38 UTC)" (scheduled at 2025-02-07 04:41:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:41:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:41:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 28, 732869), True, datetime.datetime(2025, 2, 7, 4, 41, 28, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:41:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:41:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:41:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:38 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:41:29,206 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:41:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:41:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:44.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 04:41:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:44 UTC)" (scheduled at 2025-02-07 04:41:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:41:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:41:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:41:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:41:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:41:29,235 [235] [DEBUG] [app] Starting request: urn:request:aa244332-0060-4c92-9253-773f4ea88100 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:29,235 [238] [DEBUG] [app] Starting request: urn:request:ec896497-bf23-4725-835c-d6d2fab21a9b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:41:29,238 [246] [DEBUG] [app] Starting request: urn:request:e56241ab-c2d2-4bf5-863f-97d1fe97d672 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:29,238 [239] [DEBUG] [app] Starting request: urn:request:d7f6ee22-f156-483c-9374-67aade8ebc77 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:29,238 [239] [DEBUG] [app] Ending request: urn:request:d7f6ee22-f156-483c-9374-67aade8ebc77 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d7f6ee22-f156-483c-9374-67aade8ebc77', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:41:29,238 [246] [DEBUG] [app] Ending request: urn:request:e56241ab-c2d2-4bf5-863f-97d1fe97d672 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e56241ab-c2d2-4bf5-863f-97d1fe97d672', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:41:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:41:29,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:41:29,238 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:29,241 [237] [DEBUG] [app] Starting request: urn:request:01dc1cd3-5f56-4d56-b59b-76064f5e8b6f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:29,241 [238] [DEBUG] [app] Starting request: urn:request:0c0708d4-c293-42b5-8f24-a4d3501ebae3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:29,241 [237] [DEBUG] [app] Ending request: urn:request:01dc1cd3-5f56-4d56-b59b-76064f5e8b6f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01dc1cd3-5f56-4d56-b59b-76064f5e8b6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:41:29,241 [238] [DEBUG] [app] Ending request: urn:request:0c0708d4-c293-42b5-8f24-a4d3501ebae3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0c0708d4-c293-42b5-8f24-a4d3501ebae3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:41:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:41:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:41:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:29,253 [235] [DEBUG] [app] Ending request: urn:request:aa244332-0060-4c92-9253-773f4ea88100 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aa244332-0060-4c92-9253-773f4ea88100', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:41:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:41:29,253 [238] [DEBUG] [app] Ending request: urn:request:ec896497-bf23-4725-835c-d6d2fab21a9b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec896497-bf23-4725-835c-d6d2fab21a9b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 04:41:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
repositoryactioncounter stdout | 2025-02-07 04:41:30,931 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:41:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:41:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:32.625945+00:00 (in 1.004333 seconds)
namespacegcworker stdout | 2025-02-07 04:41:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:31 UTC)" (scheduled at 2025-02-07 04:41:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:41:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:41:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 31, 621879), True, datetime.datetime(2025, 2, 7, 4, 41, 31, 621879), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:41:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:41:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:41:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:41:32,131 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:41:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:41:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:13.624301+00:00 (in 40.997914 seconds)
namespacegcworker stdout | 2025-02-07 04:41:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:32 UTC)" (scheduled at 2025-02-07 04:41:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:41:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:41:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:41:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:41:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:03.919286+00:00 (in 29.999558 seconds)
buildlogsarchiver stdout | 2025-02-07 04:41:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:03 UTC)" (scheduled at 2025-02-07 04:41:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:41:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 41, 33, 919972), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:41:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:41:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:41:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:41:36,448 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:41:37,482 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:56.713547+00:00 (in 18.997185 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:37 UTC)" (scheduled at 2025-02-07 04:41:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:41:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:41:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:48.732147+00:00 (in 9.999512 seconds)
notificationworker stdout | 2025-02-07 04:41:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:48 UTC)" (scheduled at 2025-02-07 04:41:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:41:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:41:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 38, 732913), True, datetime.datetime(2025, 2, 7, 4, 41, 38, 732913), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:41:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:41:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:41:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:41:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:41:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:54.423055+00:00 (in 13.998341 seconds)
gcworker stdout | 2025-02-07 04:41:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:10 UTC)" (scheduled at 2025-02-07 04:41:40.424250+00:00)
gcworker stdout | 2025-02-07 04:41:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:41:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903000431, None, 1, 0])
gcworker stdout | 2025-02-07 04:41:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:41:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:41:41,432 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:41:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:41:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:11.831521+00:00 (in 29.999515 seconds)
autopruneworker stdout | 2025-02-07 04:41:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:11 UTC)" (scheduled at 2025-02-07 04:41:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:41:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899701836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:41:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:41:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:41:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:41:42,066 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:41:42,530 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:41:43,857 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:41:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:41:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:59.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 04:41:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:59 UTC)" (scheduled at 2025-02-07 04:41:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:41:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:41:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:41:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:41:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:41:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:41:44,235 [237] [DEBUG] [app] Starting request: urn:request:341c9cdc-a74b-45d9-9b86-55841edff4a9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:44,235 [235] [DEBUG] [app] Starting request: urn:request:b2f84ab5-49d9-46af-9434-7cbe3d9f3a97 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:41:44,238 [244] [DEBUG] [app] Starting request: urn:request:2a6cc7c0-a3cd-4c22-9e72-a5ba788490d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:44,238 [245] [DEBUG] [app] Starting request: urn:request:9377eb45-4fdb-4944-afb3-af3f8ad66c6d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:44,238 [245] [DEBUG] [app] Ending request: urn:request:9377eb45-4fdb-4944-afb3-af3f8ad66c6d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9377eb45-4fdb-4944-afb3-af3f8ad66c6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:41:44,238 [244] [DEBUG] [app] Ending request: urn:request:2a6cc7c0-a3cd-4c22-9e72-a5ba788490d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a6cc7c0-a3cd-4c22-9e72-a5ba788490d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:41:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:41:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:41:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:44,242 [236] [DEBUG] [app] Starting request: urn:request:53fada35-381b-4ea8-8477-f8ae706e9bc0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:44,242 [238] [DEBUG] [app] Starting request: urn:request:1b4921b2-8f19-4c8d-8b1f-e030122d645b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:44,242 [236] [DEBUG] [app] Ending request: urn:request:53fada35-381b-4ea8-8477-f8ae706e9bc0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53fada35-381b-4ea8-8477-f8ae706e9bc0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:41:44,242 [238] [DEBUG] [app] Ending request: urn:request:1b4921b2-8f19-4c8d-8b1f-e030122d645b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1b4921b2-8f19-4c8d-8b1f-e030122d645b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:41:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:41:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:41:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:44,254 [235] [DEBUG] [app] Ending request: urn:request:b2f84ab5-49d9-46af-9434-7cbe3d9f3a97 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2f84ab5-49d9-46af-9434-7cbe3d9f3a97', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:41:44,254 [237] [DEBUG] [app] Ending request: urn:request:341c9cdc-a74b-45d9-9b86-55841edff4a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:341c9cdc-a74b-45d9-9b86-55841edff4a9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:41:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:41:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
notificationworker stdout | 2025-02-07 04:41:44,774 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:41:44,859 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:41:45,889 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:46,377 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:41:46,901 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:41:46,925 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-web stdout | 2025-02-07 04:41:46,962 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:41:46,965 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:41:46,968 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:41:46,970 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:41:46,973 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
exportactionlogsworker stdout | 2025-02-07 04:41:47,043 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:41:47,498 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:41:47,987 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:41:47,990 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gcworker stdout | 2025-02-07 04:41:47,992 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-secscan stdout | 2025-02-07 04:41:47,995 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
securityscanningnotificationworker stdout | 2025-02-07 04:41:48,150 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:41:48,175 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:41:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:41:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:50.814677+00:00 (in 2.082085 seconds)
notificationworker stdout | 2025-02-07 04:41:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:58 UTC)" (scheduled at 2025-02-07 04:41:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:41:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:41:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 48, 732911), True, datetime.datetime(2025, 2, 7, 4, 41, 48, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:41:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:41:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:41:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:41:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:41:49,207 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:41:49,215 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:41:49,218 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:41:49,220 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:41:49,223 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:41:49,226 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:41:49,230 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:41:49,233 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:41:49,235 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:41:49,743 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:41:49,769 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:41:50,218 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:41:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:41:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:41:58.732147+00:00 (in 7.917019 seconds)
notificationworker stdout | 2025-02-07 04:41:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:50 UTC)" (scheduled at 2025-02-07 04:41:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:41:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:41:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:50 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 04:41:52,134 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2025-02-07 04:41:52,135 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:52.134599+00:00 (in 3599.999329 seconds)
manifestbackfillworker stdout | 2025-02-07 04:41:52,135 [68] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-02-07 05:41:52 UTC)" (scheduled at 2025-02-07 04:41:52.134599+00:00)
manifestbackfillworker stdout | 2025-02-07 04:41:52,135 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0])
manifestbackfillworker stdout | 2025-02-07 04:41:52,142 [68] [DEBUG] [__main__] Manifest backfill worker has completed; skipping
manifestbackfillworker stdout | 2025-02-07 04:41:52,142 [68] [DEBUG] [data.database] Disconnecting from database.
manifestbackfillworker stdout | 2025-02-07 04:41:52,142 [68] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-02-07 05:41:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:41:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:41:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:10.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 04:41:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:24 UTC)" (scheduled at 2025-02-07 04:41:54.423055+00:00)
gcworker stdout | 2025-02-07 04:41:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:41:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:41:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:41:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:56.118855+00:00 (in 59.999516 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:41:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:56 UTC)" (scheduled at 2025-02-07 04:41:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:41:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:41:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:41:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:37.715878+00:00 (in 41.001892 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:56 UTC)" (scheduled at 2025-02-07 04:41:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:41:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:41:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:41:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:08.732147+00:00 (in 9.999574 seconds)
notificationworker stdout | 2025-02-07 04:41:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:08 UTC)" (scheduled at 2025-02-07 04:41:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:41:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:41:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 41, 58, 732856), True, datetime.datetime(2025, 2, 7, 4, 41, 58, 732856), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:41:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:41:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:41:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:08 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:41:59,218 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
quotatotalworker stdout | 2025-02-07 04:41:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:41:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:14.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 04:41:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:14 UTC)" (scheduled at 2025-02-07 04:41:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:41:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:41:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:41:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:41:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:41:59,235 [238] [DEBUG] [app] Starting request: urn:request:805966ac-1263-4f99-b0cb-f963a9121e47 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:59,235 [236] [DEBUG] [app] Starting request: urn:request:471060f2-9560-4b60-a693-5c72e342afed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:41:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:41:59,238 [239] [DEBUG] [app] Starting request: urn:request:8dfbe67b-5899-4336-8351-53f44ec01279 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:59,238 [242] [DEBUG] [app] Starting request: urn:request:8c64ec24-7b86-4787-a35a-3a54bdbbe5dc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:41:59,239 [242] [DEBUG] [app] Ending request: urn:request:8c64ec24-7b86-4787-a35a-3a54bdbbe5dc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c64ec24-7b86-4787-a35a-3a54bdbbe5dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:41:59,239 [239] [DEBUG] [app] Ending request: urn:request:8dfbe67b-5899-4336-8351-53f44ec01279 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8dfbe67b-5899-4336-8351-53f44ec01279', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:41:59,239 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:41:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:41:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:41:59,241 [236] [DEBUG] [app] Starting request: urn:request:1a62eea1-6617-4830-8361-02d7aa0b926a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:59,242 [236] [DEBUG] [app] Ending request: urn:request:1a62eea1-6617-4830-8361-02d7aa0b926a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1a62eea1-6617-4830-8361-02d7aa0b926a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:41:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:41:59,242 [237] [DEBUG] [app] Starting request: urn:request:bc5ef2e1-15b1-4724-8f81-c4373c8c98d2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:41:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:59,243 [237] [DEBUG] [app] Ending request: urn:request:bc5ef2e1-15b1-4724-8f81-c4373c8c98d2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc5ef2e1-15b1-4724-8f81-c4373c8c98d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:41:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:41:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:41:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:41:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:41:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:41:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:41:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:41:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:41:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:41:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:41:59,253 [236] [DEBUG] [app] Ending request: urn:request:471060f2-9560-4b60-a693-5c72e342afed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:471060f2-9560-4b60-a693-5c72e342afed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:59,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:41:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
gunicorn-web stdout | 2025-02-07 04:41:59,254 [238] [DEBUG] [app] Ending request: urn:request:805966ac-1263-4f99-b0cb-f963a9121e47 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:805966ac-1263-4f99-b0cb-f963a9121e47', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:41:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:41:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:41:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:41:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:41:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:41:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:04.936688+00:00 (in 4.998306 seconds)
exportactionlogsworker stdout | 2025-02-07 04:41:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:59 UTC)" (scheduled at 2025-02-07 04:41:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:41:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:41:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:42:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:42:00,944 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:42:02,167 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:42:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:42:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:23.420886+00:00 (in 20.001295 seconds)
repositorygcworker stdout | 2025-02-07 04:42:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:03 UTC)" (scheduled at 2025-02-07 04:42:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:42:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:42:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 3, 419859), True, datetime.datetime(2025, 2, 7, 4, 42, 3, 419859), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:42:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:42:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:42:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:42:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:42:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:33.919286+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2025-02-07 04:42:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:33 UTC)" (scheduled at 2025-02-07 04:42:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:42:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 42, 3, 920003), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:42:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:42:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:42:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:42:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:42:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:59.937899+00:00 (in 55.000761 seconds)
exportactionlogsworker stdout | 2025-02-07 04:42:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:04 UTC)" (scheduled at 2025-02-07 04:42:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:42:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:42:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 4, 937479), True, datetime.datetime(2025, 2, 7, 4, 42, 4, 937479), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:42:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:42:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:42:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:42:06,484 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:42:07,519 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:42:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:18.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 04:42:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:18 UTC)" (scheduled at 2025-02-07 04:42:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:42:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:42:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 8, 732927), True, datetime.datetime(2025, 2, 7, 4, 42, 8, 732927), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:42:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:42:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:42:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:42:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:42:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:24.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 04:42:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:40 UTC)" (scheduled at 2025-02-07 04:42:10.424250+00:00)
gcworker stdout | 2025-02-07 04:42:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:42:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903030430, None, 1, 0])
gcworker stdout | 2025-02-07 04:42:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:42:10,432 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:42:11,447 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:42:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:42:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:41.831521+00:00 (in 29.999520 seconds)
autopruneworker stdout | 2025-02-07 04:42:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:41 UTC)" (scheduled at 2025-02-07 04:42:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:42:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899731836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:42:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:42:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:42:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:42:12,098 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:42:12,547 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
namespacegcworker stdout | 2025-02-07 04:42:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:42:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:31.621131+00:00 (in 17.996382 seconds)
namespacegcworker stdout | 2025-02-07 04:42:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:47:13 UTC)" (scheduled at 2025-02-07 04:42:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:42:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 42, 13, 624992), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:42:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 42, 13, 624992), True, datetime.datetime(2025, 2, 7, 4, 42, 13, 624992), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:42:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 42, 13, 624992), True, datetime.datetime(2025, 2, 7, 4, 42, 13, 624992), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 42, 13, 624992), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:42:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:42:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:47:13 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:42:13,887 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:42:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:42:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:29.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 04:42:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:29 UTC)" (scheduled at 2025-02-07 04:42:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:42:14,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:42:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:42:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:42:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:42:14,235 [237] [DEBUG] [app] Starting request: urn:request:c277f13b-8bc8-41be-87ed-dce9ccbe9798 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:14,235 [235] [DEBUG] [app] Starting request: urn:request:040725c1-0213-4592-8fba-ff8a92304acf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:42:14,238 [246] [DEBUG] [app] Starting request: urn:request:a4aeb832-0590-482e-bf1f-940b52ba8167 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:14,238 [239] [DEBUG] [app] Starting request: urn:request:64c68cc8-6883-4738-9ab6-f98e7d5daf87 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:14,239 [239] [DEBUG] [app] Ending request: urn:request:64c68cc8-6883-4738-9ab6-f98e7d5daf87 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:64c68cc8-6883-4738-9ab6-f98e7d5daf87', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:42:14,239 [246] [DEBUG] [app] Ending request: urn:request:a4aeb832-0590-482e-bf1f-940b52ba8167 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4aeb832-0590-482e-bf1f-940b52ba8167', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:42:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:42:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:42:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:14,242 [238] [DEBUG] [app] Starting request: urn:request:c481cd28-1bd6-4d03-894e-69f260ed70b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:14,242 [238] [DEBUG] [app] Ending request: urn:request:c481cd28-1bd6-4d03-894e-69f260ed70b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c481cd28-1bd6-4d03-894e-69f260ed70b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:42:14,242 [235] [DEBUG] [app] Starting request: urn:request:da3665f3-0bb5-4bc4-81cb-17f494bf1917 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:14,242 [235] [DEBUG] [app] Ending request: urn:request:da3665f3-0bb5-4bc4-81cb-17f494bf1917 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:da3665f3-0bb5-4bc4-81cb-17f494bf1917', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:42:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:42:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:42:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:14,254 [235] [DEBUG] [app] Ending request: urn:request:040725c1-0213-4592-8fba-ff8a92304acf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:040725c1-0213-4592-8fba-ff8a92304acf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:42:14,254 [237] [DEBUG] [app] Ending request: urn:request:c277f13b-8bc8-41be-87ed-dce9ccbe9798 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c277f13b-8bc8-41be-87ed-dce9ccbe9798', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:42:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:42:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:42:14,792 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:42:14,893 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:42:15,918 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:16,403 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:42:16,934 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:42:16,941 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-web stdout | 2025-02-07 04:42:16,968 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:42:16,971 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:42:16,974 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:42:16,978 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:42:16,980 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
exportactionlogsworker stdout | 2025-02-07 04:42:17,079 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:42:17,505 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:42:17,995 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:42:17,999 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:42:18,001 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:42:18,028 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:42:18,179 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:42:18,211 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:42:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:28.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 04:42:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:28 UTC)" (scheduled at 2025-02-07 04:42:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:42:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:42:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 18, 732878), True, datetime.datetime(2025, 2, 7, 4, 42, 18, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:42:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:42:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:42:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:42:19,215 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:42:19,224 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:42:19,228 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:42:19,231 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:42:19,235 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:42:19,238 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:42:19,240 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:42:19,243 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:42:19,245 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:42:19,756 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:42:19,805 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:42:20,230 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:42:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:42:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:03.419110+00:00 (in 39.997752 seconds)
repositorygcworker stdout | 2025-02-07 04:42:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:23 UTC)" (scheduled at 2025-02-07 04:42:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:42:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:42:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:42:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:42:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:40.424250+00:00 (in 16.000740 seconds)
gcworker stdout | 2025-02-07 04:42:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:54 UTC)" (scheduled at 2025-02-07 04:42:24.423055+00:00)
gcworker stdout | 2025-02-07 04:42:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:42:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:42:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:42:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:33.814166+00:00 (in 5.081575 seconds)
notificationworker stdout | 2025-02-07 04:42:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:38 UTC)" (scheduled at 2025-02-07 04:42:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:42:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:42:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 28, 732865), True, datetime.datetime(2025, 2, 7, 4, 42, 28, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:42:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:42:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:42:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:42:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:42:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:44.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:42:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:44 UTC)" (scheduled at 2025-02-07 04:42:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:42:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:42:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:42:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:42:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:42:29,235 [235] [DEBUG] [app] Starting request: urn:request:a9e6b140-7e7d-4cc5-aaab-4f5ab698733a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:29,235 [238] [DEBUG] [app] Starting request: urn:request:24cbbc22-8746-421f-a8f8-a023829c5741 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:42:29,238 [245] [DEBUG] [app] Starting request: urn:request:5158f44a-be35-409c-88a2-3a827bcd9adf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:29,238 [239] [DEBUG] [app] Starting request: urn:request:3d93cac5-e770-495e-927c-90f977547642 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:29,238 [245] [DEBUG] [app] Ending request: urn:request:5158f44a-be35-409c-88a2-3a827bcd9adf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5158f44a-be35-409c-88a2-3a827bcd9adf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:42:29,238 [239] [DEBUG] [app] Ending request: urn:request:3d93cac5-e770-495e-927c-90f977547642 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d93cac5-e770-495e-927c-90f977547642', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:42:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:42:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:42:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:29,241 [238] [DEBUG] [app] Starting request: urn:request:a6694e2d-29e5-4e9b-baaf-48481c10b704 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:29,241 [238] [DEBUG] [app] Ending request: urn:request:a6694e2d-29e5-4e9b-baaf-48481c10b704 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6694e2d-29e5-4e9b-baaf-48481c10b704', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:42:29,241 [236] [DEBUG] [app] Starting request: urn:request:5d05af43-f9cd-4c9c-a9ce-5655b971c0ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:29,242 [236] [DEBUG] [app] Ending request: urn:request:5d05af43-f9cd-4c9c-a9ce-5655b971c0ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5d05af43-f9cd-4c9c-a9ce-5655b971c0ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:42:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:42:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:42:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:42:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:29,253 [238] [DEBUG] [app] Ending request: urn:request:24cbbc22-8746-421f-a8f8-a023829c5741 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:24cbbc22-8746-421f-a8f8-a023829c5741', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
storagereplication stdout | 2025-02-07 04:42:29,253 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
gunicorn-web stdout | 2025-02-07 04:42:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 04:42:29,254 [235] [DEBUG] [app] Ending request: urn:request:a9e6b140-7e7d-4cc5-aaab-4f5ab698733a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a9e6b140-7e7d-4cc5-aaab-4f5ab698733a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:42:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
repositoryactioncounter stdout | 2025-02-07 04:42:30,970 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:42:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:42:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:32.625945+00:00 (in 1.004375 seconds)
namespacegcworker stdout | 2025-02-07 04:42:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:31 UTC)" (scheduled at 2025-02-07 04:42:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:42:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:42:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 31, 621843), True, datetime.datetime(2025, 2, 7, 4, 42, 31, 621843), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:42:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:42:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:42:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:42:32,203 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:42:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:42:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:31.621131+00:00 (in 58.994743 seconds)
namespacegcworker stdout | 2025-02-07 04:42:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:32 UTC)" (scheduled at 2025-02-07 04:42:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:42:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:42:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:42:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:38.732147+00:00 (in 4.917530 seconds)
notificationworker stdout | 2025-02-07 04:42:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:47:33 UTC)" (scheduled at 2025-02-07 04:42:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:42:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 42, 33, 814858), 'notification/%'])
notificationworker stdout | 2025-02-07 04:42:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 42, 33, 814858), True, datetime.datetime(2025, 2, 7, 4, 42, 33, 814858), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:42:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 42, 33, 814858), True, datetime.datetime(2025, 2, 7, 4, 42, 33, 814858), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 42, 33, 814858), 'notification/%'])
notificationworker stdout | 2025-02-07 04:42:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:42:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:47:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:42:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:42:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:03.919286+00:00 (in 29.999560 seconds)
buildlogsarchiver stdout | 2025-02-07 04:42:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:03 UTC)" (scheduled at 2025-02-07 04:42:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:42:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 42, 33, 919973), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:42:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:42:33,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:42:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:42:36,519 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:42:37,527 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:56.713547+00:00 (in 18.997195 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:37 UTC)" (scheduled at 2025-02-07 04:42:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:42:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:48.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 04:42:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:48 UTC)" (scheduled at 2025-02-07 04:42:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:42:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:42:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 38, 732840), True, datetime.datetime(2025, 2, 7, 4, 42, 38, 732840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:42:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:42:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:42:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:42:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:42:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:54.423055+00:00 (in 13.998317 seconds)
gcworker stdout | 2025-02-07 04:42:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:10 UTC)" (scheduled at 2025-02-07 04:42:40.424250+00:00)
gcworker stdout | 2025-02-07 04:42:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:42:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903060431, None, 1, 0])
gcworker stdout | 2025-02-07 04:42:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:42:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:42:41,483 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:42:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:42:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:11.831521+00:00 (in 29.999577 seconds)
autopruneworker stdout | 2025-02-07 04:42:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:11 UTC)" (scheduled at 2025-02-07 04:42:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:42:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899761836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:42:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:42:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:42:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:42:42,122 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:42:42,559 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:42:43,914 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:42:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:42:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:59.225378+00:00 (in 14.999579 seconds)
quotatotalworker stdout | 2025-02-07 04:42:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:59 UTC)" (scheduled at 2025-02-07 04:42:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:42:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:42:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:42:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:42:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:42:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:42:44,235 [235] [DEBUG] [app] Starting request: urn:request:45c78dda-8845-4ced-8078-84cea65ed65b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:44,235 [238] [DEBUG] [app] Starting request: urn:request:9a8ec79d-81ed-4d7f-8bb9-081660c89fe5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:42:44,238 [246] [DEBUG] [app] Starting request: urn:request:3d705ebd-e7e9-4631-b84d-b3a4c496a248 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:44,238 [239] [DEBUG] [app] Starting request: urn:request:5cf2f087-aed9-42a2-b1a8-f18f9a487ccb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:44,238 [246] [DEBUG] [app] Ending request: urn:request:3d705ebd-e7e9-4631-b84d-b3a4c496a248 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d705ebd-e7e9-4631-b84d-b3a4c496a248', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:42:44,239 [239] [DEBUG] [app] Ending request: urn:request:5cf2f087-aed9-42a2-b1a8-f18f9a487ccb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cf2f087-aed9-42a2-b1a8-f18f9a487ccb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:42:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:42:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:42:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:44,241 [236] [DEBUG] [app] Starting request: urn:request:97b19b65-0fb1-404d-8333-47867f9c6f32 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:44,242 [236] [DEBUG] [app] Ending request: urn:request:97b19b65-0fb1-404d-8333-47867f9c6f32 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:97b19b65-0fb1-404d-8333-47867f9c6f32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:42:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:44,242 [235] [DEBUG] [app] Starting request: urn:request:61dc25e7-d57e-4964-a9cf-41a6a1553ed4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:44,243 [235] [DEBUG] [app] Ending request: urn:request:61dc25e7-d57e-4964-a9cf-41a6a1553ed4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61dc25e7-d57e-4964-a9cf-41a6a1553ed4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:42:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:42:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:42:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:44,254 [235] [DEBUG] [app] Ending request: urn:request:45c78dda-8845-4ced-8078-84cea65ed65b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:45c78dda-8845-4ced-8078-84cea65ed65b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:44,254 [238] [DEBUG] [app] Ending request: urn:request:9a8ec79d-81ed-4d7f-8bb9-081660c89fe5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a8ec79d-81ed-4d7f-8bb9-081660c89fe5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:42:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:42:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:42:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
notificationworker stdout | 2025-02-07 04:42:44,804 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:42:44,924 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:42:45,942 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:46,436 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 04:42:46,971 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:42:46,974 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-web stdout | 2025-02-07 04:42:46,977 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:42:46,980 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:42:46,984 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:42:46,986 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:42:46,989 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
exportactionlogsworker stdout | 2025-02-07 04:42:47,116 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:42:47,541 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:42:48,002 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:42:48,005 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:42:48,009 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:42:48,034 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:42:48,206 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:42:48,247 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:42:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:50.814677+00:00 (in 2.082105 seconds)
notificationworker stdout | 2025-02-07 04:42:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:58 UTC)" (scheduled at 2025-02-07 04:42:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:42:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:42:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 48, 732838), True, datetime.datetime(2025, 2, 7, 4, 42, 48, 732838), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:42:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:42:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:42:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:42:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:42:49,228 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:42:49,234 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:42:49,237 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:42:49,241 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:42:49,245 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:42:49,248 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:42:49,251 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:42:49,255 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:42:49,257 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:42:49,792 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:42:49,840 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:42:50,262 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:42:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:42:58.732147+00:00 (in 7.917033 seconds)
notificationworker stdout | 2025-02-07 04:42:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:50 UTC)" (scheduled at 2025-02-07 04:42:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:42:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:42:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:42:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:42:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:10.424250+00:00 (in 16.000707 seconds)
gcworker stdout | 2025-02-07 04:42:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:24 UTC)" (scheduled at 2025-02-07 04:42:54.423055+00:00)
gcworker stdout | 2025-02-07 04:42:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:42:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:42:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:42:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:56.118855+00:00 (in 59.999502 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:42:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:56 UTC)" (scheduled at 2025-02-07 04:42:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:42:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:42:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:42:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:37.715878+00:00 (in 41.001881 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:56 UTC)" (scheduled at 2025-02-07 04:42:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:42:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:42:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:42:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:08.732147+00:00 (in 9.999575 seconds)
notificationworker stdout | 2025-02-07 04:42:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:08 UTC)" (scheduled at 2025-02-07 04:42:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:42:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:42:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 42, 58, 732838), True, datetime.datetime(2025, 2, 7, 4, 42, 58, 732838), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:42:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:42:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:42:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:42:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:42:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:14.225378+00:00 (in 14.999569 seconds)
quotatotalworker stdout | 2025-02-07 04:42:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:14 UTC)" (scheduled at 2025-02-07 04:42:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:42:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:42:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:42:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:42:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:42:59,235 [236] [DEBUG] [app] Starting request: urn:request:558ef415-1216-4775-aea0-4aa2f75ee514 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:59,235 [235] [DEBUG] [app] Starting request: urn:request:860c46af-b051-41df-a0d5-82cb0257eaf6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:42:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:42:59,238 [239] [DEBUG] [app] Starting request: urn:request:4889ea6b-bc58-4e83-b814-a51a03db1329 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:59,238 [246] [DEBUG] [app] Starting request: urn:request:4709f958-a8fd-4e71-bcf7-83b824cfe7a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:42:59,239 [239] [DEBUG] [app] Ending request: urn:request:4889ea6b-bc58-4e83-b814-a51a03db1329 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4889ea6b-bc58-4e83-b814-a51a03db1329', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:42:59,239 [246] [DEBUG] [app] Ending request: urn:request:4709f958-a8fd-4e71-bcf7-83b824cfe7a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4709f958-a8fd-4e71-bcf7-83b824cfe7a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:42:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:42:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:42:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:42:59,241 [236] [DEBUG] [app] Starting request: urn:request:0a6bc676-8433-4947-8189-70145bd1069c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:59,241 [238] [DEBUG] [app] Starting request: urn:request:50255d92-1a67-420d-ae79-c22b3137e29a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:42:59,242 [236] [DEBUG] [app] Ending request: urn:request:0a6bc676-8433-4947-8189-70145bd1069c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a6bc676-8433-4947-8189-70145bd1069c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:42:59,242 [238] [DEBUG] [app] Ending request: urn:request:50255d92-1a67-420d-ae79-c22b3137e29a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50255d92-1a67-420d-ae79-c22b3137e29a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:42:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:42:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:42:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:42:59,242 [236] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:42:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:42:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:42:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:42:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:42:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:42:59,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:42:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:42:59,253 [236] [DEBUG] [app] Ending request: urn:request:558ef415-1216-4775-aea0-4aa2f75ee514 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:558ef415-1216-4775-aea0-4aa2f75ee514', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:59,253 [235] [DEBUG] [app] Ending request: urn:request:860c46af-b051-41df-a0d5-82cb0257eaf6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:860c46af-b051-41df-a0d5-82cb0257eaf6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:42:59,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:42:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:42:59,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:42:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:42:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:42:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
storagereplication stdout | 2025-02-07 04:42:59,274 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:42:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:42:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:04.936688+00:00 (in 4.998354 seconds)
exportactionlogsworker stdout | 2025-02-07 04:42:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:59 UTC)" (scheduled at 2025-02-07 04:42:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:42:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:42:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:43:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:43:01,006 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:43:02,231 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:43:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:43:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:23.420886+00:00 (in 20.001307 seconds)
repositorygcworker stdout | 2025-02-07 04:43:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:03 UTC)" (scheduled at 2025-02-07 04:43:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:43:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:43:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 3, 419848), True, datetime.datetime(2025, 2, 7, 4, 43, 3, 419848), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:43:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:43:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:43:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:43:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:43:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:33.919286+00:00 (in 29.999543 seconds)
buildlogsarchiver stdout | 2025-02-07 04:43:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:33 UTC)" (scheduled at 2025-02-07 04:43:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:43:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 43, 3, 919998), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:43:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:43:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:43:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:43:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:43:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:59.937899+00:00 (in 55.000790 seconds)
exportactionlogsworker stdout | 2025-02-07 04:43:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:04 UTC)" (scheduled at 2025-02-07 04:43:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:43:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:43:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 4, 937406), True, datetime.datetime(2025, 2, 7, 4, 43, 4, 937406), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:43:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:43:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:43:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:43:06,525 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:43:07,563 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:43:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:43:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:18.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 04:43:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:18 UTC)" (scheduled at 2025-02-07 04:43:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:43:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:43:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 8, 732848), True, datetime.datetime(2025, 2, 7, 4, 43, 8, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:43:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:43:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:43:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:43:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:43:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:24.423055+00:00 (in 13.998384 seconds)
gcworker stdout | 2025-02-07 04:43:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:40 UTC)" (scheduled at 2025-02-07 04:43:10.424250+00:00)
gcworker stdout | 2025-02-07 04:43:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:43:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903090431, None, 1, 0])
gcworker stdout | 2025-02-07 04:43:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:43:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:43:11,507 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:43:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:43:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:41.831521+00:00 (in 29.999558 seconds)
autopruneworker stdout | 2025-02-07 04:43:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:41 UTC)" (scheduled at 2025-02-07 04:43:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:43:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899791836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:43:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:43:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:43:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:43:12,146 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:43:12,579 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:43:13,935 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:43:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:43:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:29.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 04:43:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:29 UTC)" (scheduled at 2025-02-07 04:43:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:43:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:43:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:43:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:43:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:43:14,235 [235] [DEBUG] [app] Starting request: urn:request:785a3bf4-6637-475a-a539-5c683dad8855 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:14,235 [238] [DEBUG] [app] Starting request: urn:request:26da9d65-9870-4a2c-bab3-26a856635348 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:43:14,238 [246] [DEBUG] [app] Starting request: urn:request:97b1bf68-c68d-422f-85ae-78d6d478952c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:14,238 [239] [DEBUG] [app] Starting request: urn:request:f9dee86a-ed06-4422-82e1-09da03e3e78c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:14,239 [246] [DEBUG] [app] Ending request: urn:request:97b1bf68-c68d-422f-85ae-78d6d478952c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97b1bf68-c68d-422f-85ae-78d6d478952c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:43:14,239 [239] [DEBUG] [app] Ending request: urn:request:f9dee86a-ed06-4422-82e1-09da03e3e78c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9dee86a-ed06-4422-82e1-09da03e3e78c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:43:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:43:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:14,242 [235] [DEBUG] [app] Starting request: urn:request:7254afcf-b50d-4e82-ab23-7138d304615c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:43:14,242 [237] [DEBUG] [app] Starting request: urn:request:a2281a4c-a4a7-4e78-8581-7e957bc9d1a9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:43:14,242 [235] [DEBUG] [app] Ending request: urn:request:7254afcf-b50d-4e82-ab23-7138d304615c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7254afcf-b50d-4e82-ab23-7138d304615c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:43:14,242 [237] [DEBUG] [app] Ending request: urn:request:a2281a4c-a4a7-4e78-8581-7e957bc9d1a9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a2281a4c-a4a7-4e78-8581-7e957bc9d1a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:43:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:43:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:43:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:43:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:43:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:43:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:14,254 [235] [DEBUG] [app] Ending request: urn:request:785a3bf4-6637-475a-a539-5c683dad8855 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:785a3bf4-6637-475a-a539-5c683dad8855', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:43:14,254 [238] [DEBUG] [app] Ending request: urn:request:26da9d65-9870-4a2c-bab3-26a856635348 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26da9d65-9870-4a2c-bab3-26a856635348', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:43:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:43:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:43:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:43:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
notificationworker stdout | 2025-02-07 04:43:14,840 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:43:14,931 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:43:15,972 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:16,472 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:43:16,985 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:43:16,988 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:43:16,991 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:43:16,994 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
servicekey stdout | 2025-02-07 04:43:16,997 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-web stdout | 2025-02-07 04:43:17,000 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:43:17,009 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:43:17,129 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:43:17,577 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:43:18,011 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:43:18,014 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:43:18,017 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:43:18,062 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:43:18,218 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:43:18,283 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:43:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:43:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:28.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 04:43:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:28 UTC)" (scheduled at 2025-02-07 04:43:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:43:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:43:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 18, 732890), True, datetime.datetime(2025, 2, 7, 4, 43, 18, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:43:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:43:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:43:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:43:19,240 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:43:19,243 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:43:19,246 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:43:19,249 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:43:19,253 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:43:19,255 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:43:19,259 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:43:19,262 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:43:19,264 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:43:19,829 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:43:19,877 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:43:20,294 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:43:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:43:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:32.420707+00:00 (in 8.999331 seconds)
repositorygcworker stdout | 2025-02-07 04:43:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:23 UTC)" (scheduled at 2025-02-07 04:43:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:43:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:43:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:43:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:43:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:40.424250+00:00 (in 16.000746 seconds)
gcworker stdout | 2025-02-07 04:43:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:54 UTC)" (scheduled at 2025-02-07 04:43:24.423055+00:00)
gcworker stdout | 2025-02-07 04:43:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:43:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:43:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:43:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:43:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:38.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 04:43:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:38 UTC)" (scheduled at 2025-02-07 04:43:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:43:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:43:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 28, 732864), True, datetime.datetime(2025, 2, 7, 4, 43, 28, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:43:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:43:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:43:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:43:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:43:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:44.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 04:43:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:44 UTC)" (scheduled at 2025-02-07 04:43:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:43:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:43:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:43:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:43:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:43:29,235 [238] [DEBUG] [app] Starting request: urn:request:6f039b0e-ed33-40a8-af94-633c8ee23e8b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:29,235 [235] [DEBUG] [app] Starting request: urn:request:61f173c2-0104-403c-af0c-b5c5ebfe2f88 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:43:29,238 [246] [DEBUG] [app] Starting request: urn:request:d828b303-0e7d-44c5-944e-f19597c4866b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:29,238 [239] [DEBUG] [app] Starting request: urn:request:c1a1dc82-c82d-4141-b0dd-af41d924eb1a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:29,239 [239] [DEBUG] [app] Ending request: urn:request:c1a1dc82-c82d-4141-b0dd-af41d924eb1a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c1a1dc82-c82d-4141-b0dd-af41d924eb1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:43:29,239 [246] [DEBUG] [app] Ending request: urn:request:d828b303-0e7d-44c5-944e-f19597c4866b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d828b303-0e7d-44c5-944e-f19597c4866b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:43:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:43:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:29,241 [235] [DEBUG] [app] Starting request: urn:request:4eacbc04-8cc7-4065-bd1f-bc7458213a36 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:43:29,242 [235] [DEBUG] [app] Ending request: urn:request:4eacbc04-8cc7-4065-bd1f-bc7458213a36 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4eacbc04-8cc7-4065-bd1f-bc7458213a36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:43:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:29,242 [237] [DEBUG] [app] Starting request: urn:request:4e2cb26d-c022-467a-ab8f-4cad64afbfaf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:43:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:29,242 [237] [DEBUG] [app] Ending request: urn:request:4e2cb26d-c022-467a-ab8f-4cad64afbfaf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4e2cb26d-c022-467a-ab8f-4cad64afbfaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:43:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:43:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:43:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:43:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:43:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:29,254 [235] [DEBUG] [app] Ending request: urn:request:61f173c2-0104-403c-af0c-b5c5ebfe2f88 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61f173c2-0104-403c-af0c-b5c5ebfe2f88', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:43:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:43:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:43:29,254 [238] [DEBUG] [app] Ending request: urn:request:6f039b0e-ed33-40a8-af94-633c8ee23e8b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f039b0e-ed33-40a8-af94-633c8ee23e8b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:43:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:43:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 04:43:29,310 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:43:31,043 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:43:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:43:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:32.625945+00:00 (in 1.004349 seconds)
namespacegcworker stdout | 2025-02-07 04:43:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:31 UTC)" (scheduled at 2025-02-07 04:43:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:43:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:43:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 31, 621880), True, datetime.datetime(2025, 2, 7, 4, 43, 31, 621880), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:43:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:43:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:43:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:43:32,267 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:43:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:43:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:03.419110+00:00 (in 30.997948 seconds)
repositorygcworker stdout | 2025-02-07 04:43:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:48:32 UTC)" (scheduled at 2025-02-07 04:43:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:43:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 43, 32, 421432), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:43:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 43, 32, 421432), True, datetime.datetime(2025, 2, 7, 4, 43, 32, 421432), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:43:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 43, 32, 421432), True, datetime.datetime(2025, 2, 7, 4, 43, 32, 421432), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 43, 32, 421432), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:43:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:43:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:48:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:43:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:43:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:31.621131+00:00 (in 58.994740 seconds)
namespacegcworker stdout | 2025-02-07 04:43:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:32 UTC)" (scheduled at 2025-02-07 04:43:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:43:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:43:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:43:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:43:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:03.919286+00:00 (in 29.999482 seconds)
buildlogsarchiver stdout | 2025-02-07 04:43:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:03 UTC)" (scheduled at 2025-02-07 04:43:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:43:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 43, 33, 920098), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:43:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:43:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:43:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:43:36,561 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:43:37,578 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:56.713547+00:00 (in 18.997208 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:37 UTC)" (scheduled at 2025-02-07 04:43:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:43:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:43:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:48.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 04:43:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:48 UTC)" (scheduled at 2025-02-07 04:43:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:43:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:43:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 38, 732873), True, datetime.datetime(2025, 2, 7, 4, 43, 38, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:43:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:43:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:43:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:43:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:43:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:54.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 04:43:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:10 UTC)" (scheduled at 2025-02-07 04:43:40.424250+00:00)
gcworker stdout | 2025-02-07 04:43:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:43:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903120431, None, 1, 0])
gcworker stdout | 2025-02-07 04:43:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:43:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:43:41,531 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:43:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:43:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:11.831521+00:00 (in 29.999535 seconds)
autopruneworker stdout | 2025-02-07 04:43:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:11 UTC)" (scheduled at 2025-02-07 04:43:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:43:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899821836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:43:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:43:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:43:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:43:42,183 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:43:42,591 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:43:43,964 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:43:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:43:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:59.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:43:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:59 UTC)" (scheduled at 2025-02-07 04:43:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:43:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:43:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:43:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:43:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:43:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:43:44,235 [237] [DEBUG] [app] Starting request: urn:request:6d70ea10-253d-440f-b6f7-a1504b808051 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:44,235 [236] [DEBUG] [app] Starting request: urn:request:cade2f5d-54c6-4321-a588-6e76440ce2ae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:43:44,238 [244] [DEBUG] [app] Starting request: urn:request:755923e3-319b-4b72-9a78-3aa8dfe6c978 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:44,238 [239] [DEBUG] [app] Starting request: urn:request:f3d34438-1d00-44ba-a354-18ee6035ed36 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:44,238 [244] [DEBUG] [app] Ending request: urn:request:755923e3-319b-4b72-9a78-3aa8dfe6c978 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:755923e3-319b-4b72-9a78-3aa8dfe6c978', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:43:44,238 [239] [DEBUG] [app] Ending request: urn:request:f3d34438-1d00-44ba-a354-18ee6035ed36 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3d34438-1d00-44ba-a354-18ee6035ed36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:43:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:43:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:44,241 [236] [DEBUG] [app] Starting request: urn:request:5d753e86-0421-4fe1-a695-21862a6bcf7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:43:44,241 [235] [DEBUG] [app] Starting request: urn:request:64416b72-eade-40aa-af41-51644cce13cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:43:44,241 [236] [DEBUG] [app] Ending request: urn:request:5d753e86-0421-4fe1-a695-21862a6bcf7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5d753e86-0421-4fe1-a695-21862a6bcf7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:43:44,241 [235] [DEBUG] [app] Ending request: urn:request:64416b72-eade-40aa-af41-51644cce13cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64416b72-eade-40aa-af41-51644cce13cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:43:44,241 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.000)
gunicorn-web stdout | 2025-02-07 04:43:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:43:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:44,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:43:44,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:43:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:43:44,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:43:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:44,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:44,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:44,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:44,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:44,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:44,253 [236] [DEBUG] [app] Ending request: urn:request:cade2f5d-54c6-4321-a588-6e76440ce2ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cade2f5d-54c6-4321-a588-6e76440ce2ae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:43:44,253 [237] [DEBUG] [app] Ending request: urn:request:6d70ea10-253d-440f-b6f7-a1504b808051 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6d70ea10-253d-440f-b6f7-a1504b808051', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:43:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:43:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:43:44,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:43:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
notificationworker stdout | 2025-02-07 04:43:44,854 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:43:44,944 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:43:46,008 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:46,482 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:43:46,993 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:43:46,996 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:43:46,999 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:43:47,002 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:43:47,006 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:43:47,016 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:43:47,018 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:43:47,140 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:43:47,595 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:43:48,020 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:43:48,022 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:43:48,025 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:43:48,078 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:43:48,230 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:43:48,317 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:43:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:43:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:50.814677+00:00 (in 2.082097 seconds)
notificationworker stdout | 2025-02-07 04:43:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:58 UTC)" (scheduled at 2025-02-07 04:43:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:43:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:43:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 48, 732842), True, datetime.datetime(2025, 2, 7, 4, 43, 48, 732842), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:43:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:43:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:43:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:43:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:43:49,252 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:43:49,255 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:43:49,257 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:43:49,260 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:43:49,263 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:43:49,266 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:43:49,269 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:43:49,271 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:43:49,274 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:43:49,865 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:43:49,913 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:43:50,330 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:43:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:43:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:43:58.732147+00:00 (in 7.916975 seconds)
notificationworker stdout | 2025-02-07 04:43:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:50 UTC)" (scheduled at 2025-02-07 04:43:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:43:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:43:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:43:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:43:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:10.424250+00:00 (in 16.000733 seconds)
gcworker stdout | 2025-02-07 04:43:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:24 UTC)" (scheduled at 2025-02-07 04:43:54.423055+00:00)
gcworker stdout | 2025-02-07 04:43:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:43:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:43:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:43:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:56.118855+00:00 (in 59.999511 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:43:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:56 UTC)" (scheduled at 2025-02-07 04:43:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:43:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:43:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:43:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:37.715878+00:00 (in 41.001876 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:56 UTC)" (scheduled at 2025-02-07 04:43:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:43:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:43:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:43:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:08.732147+00:00 (in 9.999487 seconds)
notificationworker stdout | 2025-02-07 04:43:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:08 UTC)" (scheduled at 2025-02-07 04:43:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:43:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:43:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 43, 58, 732938), True, datetime.datetime(2025, 2, 7, 4, 43, 58, 732938), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:43:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:43:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:43:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:43:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:43:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:14.225378+00:00 (in 14.999535 seconds)
quotatotalworker stdout | 2025-02-07 04:43:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:14 UTC)" (scheduled at 2025-02-07 04:43:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:43:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:43:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:43:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:43:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:43:59,235 [238] [DEBUG] [app] Starting request: urn:request:c272e50f-17d5-41c1-a94a-ee098c1dcc3d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:59,235 [235] [DEBUG] [app] Starting request: urn:request:0f631a66-9036-4aad-8385-e6ebbbe73b74 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:43:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:43:59,238 [246] [DEBUG] [app] Starting request: urn:request:163afc42-7d2a-45f7-8357-e373c1dea281 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:59,238 [245] [DEBUG] [app] Starting request: urn:request:407aba41-9e7e-4b33-9893-1a8ae638b927 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:43:59,238 [246] [DEBUG] [app] Ending request: urn:request:163afc42-7d2a-45f7-8357-e373c1dea281 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:163afc42-7d2a-45f7-8357-e373c1dea281', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:43:59,238 [245] [DEBUG] [app] Ending request: urn:request:407aba41-9e7e-4b33-9893-1a8ae638b927 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:407aba41-9e7e-4b33-9893-1a8ae638b927', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:43:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:43:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:43:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:43:59,241 [235] [DEBUG] [app] Starting request: urn:request:f1884aec-5599-4224-8a05-0b8eedab30ac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:43:59,241 [235] [DEBUG] [app] Ending request: urn:request:f1884aec-5599-4224-8a05-0b8eedab30ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f1884aec-5599-4224-8a05-0b8eedab30ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:43:59,241 [236] [DEBUG] [app] Starting request: urn:request:6850b0e1-e076-4920-a5ea-dc8f159624e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:43:59,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:59,242 [236] [DEBUG] [app] Ending request: urn:request:6850b0e1-e076-4920-a5ea-dc8f159624e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6850b0e1-e076-4920-a5ea-dc8f159624e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:43:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:43:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:43:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:43:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:43:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:43:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:43:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:43:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:43:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:43:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:43:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:43:59,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:43:59,253 [235] [DEBUG] [app] Ending request: urn:request:0f631a66-9036-4aad-8385-e6ebbbe73b74 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f631a66-9036-4aad-8385-e6ebbbe73b74', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:43:59,253 [238] [DEBUG] [app] Ending request: urn:request:c272e50f-17d5-41c1-a94a-ee098c1dcc3d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c272e50f-17d5-41c1-a94a-ee098c1dcc3d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:43:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:43:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:43:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:43:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:43:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:43:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
storagereplication stdout | 2025-02-07 04:43:59,347 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:43:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:43:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:04.936688+00:00 (in 4.998338 seconds)
exportactionlogsworker stdout | 2025-02-07 04:43:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:59 UTC)" (scheduled at 2025-02-07 04:43:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:43:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:43:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:44:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:44:01,078 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:44:02,275 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:44:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:44:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:23.420886+00:00 (in 20.001347 seconds)
repositorygcworker stdout | 2025-02-07 04:44:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:03 UTC)" (scheduled at 2025-02-07 04:44:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:44:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:44:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 3, 419812), True, datetime.datetime(2025, 2, 7, 4, 44, 3, 419812), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:44:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:44:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:44:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:44:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:44:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:33.919286+00:00 (in 29.999495 seconds)
buildlogsarchiver stdout | 2025-02-07 04:44:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:33 UTC)" (scheduled at 2025-02-07 04:44:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:44:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 44, 3, 920068), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:44:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:44:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:44:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:44:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:44:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:52.937648+00:00 (in 48.000532 seconds)
exportactionlogsworker stdout | 2025-02-07 04:44:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:04 UTC)" (scheduled at 2025-02-07 04:44:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:44:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:44:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 4, 937458), True, datetime.datetime(2025, 2, 7, 4, 44, 4, 937458), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:44:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:44:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:44:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:44:06,568 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:44:07,614 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:44:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:44:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:18.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 04:44:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:18 UTC)" (scheduled at 2025-02-07 04:44:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:44:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:44:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 8, 732920), True, datetime.datetime(2025, 2, 7, 4, 44, 8, 732920), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:44:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:44:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:44:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:44:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:44:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:24.423055+00:00 (in 13.998356 seconds)
gcworker stdout | 2025-02-07 04:44:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:40 UTC)" (scheduled at 2025-02-07 04:44:10.424250+00:00)
gcworker stdout | 2025-02-07 04:44:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:44:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903150430, None, 1, 0])
gcworker stdout | 2025-02-07 04:44:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:44:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:44:11,567 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:44:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:44:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:41.831521+00:00 (in 29.999459 seconds)
autopruneworker stdout | 2025-02-07 04:44:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:41 UTC)" (scheduled at 2025-02-07 04:44:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:44:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899851836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:44:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:44:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:44:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:44:12,219 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:44:12,627 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:44:13,986 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:44:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:44:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:29.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 04:44:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:29 UTC)" (scheduled at 2025-02-07 04:44:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:44:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:44:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:44:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:44:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:44:14,235 [237] [DEBUG] [app] Starting request: urn:request:3ff85052-79f6-47b8-b8d7-5e39ec3fb5c9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:14,235 [238] [DEBUG] [app] Starting request: urn:request:538f2b37-b936-4b95-8f67-8e4dfd2fb2b5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:44:14,238 [239] [DEBUG] [app] Starting request: urn:request:0387f498-4a91-47b5-8bcf-63892cf2439f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:14,239 [245] [DEBUG] [app] Starting request: urn:request:3284c1fd-3d6f-4c58-bd03-8245d4866792 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:14,239 [239] [DEBUG] [app] Ending request: urn:request:0387f498-4a91-47b5-8bcf-63892cf2439f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0387f498-4a91-47b5-8bcf-63892cf2439f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:44:14,239 [245] [DEBUG] [app] Ending request: urn:request:3284c1fd-3d6f-4c58-bd03-8245d4866792 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3284c1fd-3d6f-4c58-bd03-8245d4866792', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:44:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:44:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:44:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:14,242 [237] [DEBUG] [app] Starting request: urn:request:1cbce77b-78a7-4577-8301-3143206b4107 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:14,242 [237] [DEBUG] [app] Ending request: urn:request:1cbce77b-78a7-4577-8301-3143206b4107 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1cbce77b-78a7-4577-8301-3143206b4107', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:44:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:44:14,242 [235] [DEBUG] [app] Starting request: urn:request:4a5839ab-fccc-46a0-b443-bc75ea7f4166 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:14,243 [235] [DEBUG] [app] Ending request: urn:request:4a5839ab-fccc-46a0-b443-bc75ea7f4166 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a5839ab-fccc-46a0-b443-bc75ea7f4166', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:44:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:44:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:14,254 [237] [DEBUG] [app] Ending request: urn:request:3ff85052-79f6-47b8-b8d7-5e39ec3fb5c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ff85052-79f6-47b8-b8d7-5e39ec3fb5c9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:44:14,254 [238] [DEBUG] [app] Ending request: urn:request:538f2b37-b936-4b95-8f67-8e4dfd2fb2b5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:538f2b37-b936-4b95-8f67-8e4dfd2fb2b5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:44:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:44:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
notificationworker stdout | 2025-02-07 04:44:14,875 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:44:14,977 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:44:16,045 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:16,518 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:44:17,002 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:44:17,005 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:44:17,008 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:44:17,011 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:44:17,014 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:44:17,052 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:44:17,055 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:44:17,169 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:44:17,629 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:44:18,028 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:44:18,032 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:44:18,034 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:44:18,114 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:44:18,266 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:44:18,343 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:44:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:44:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:28.732147+00:00 (in 9.999529 seconds)
notificationworker stdout | 2025-02-07 04:44:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:28 UTC)" (scheduled at 2025-02-07 04:44:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:44:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:44:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 18, 732891), True, datetime.datetime(2025, 2, 7, 4, 44, 18, 732891), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:44:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:44:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:44:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:44:19,262 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:44:19,267 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:44:19,269 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:44:19,272 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:44:19,275 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:44:19,277 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:44:19,280 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:44:19,283 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:44:19,286 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:44:19,901 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:44:19,949 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:44:20,366 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:44:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:44:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:03.419110+00:00 (in 39.997753 seconds)
repositorygcworker stdout | 2025-02-07 04:44:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:23 UTC)" (scheduled at 2025-02-07 04:44:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:44:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:44:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:44:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:44:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:40.424250+00:00 (in 16.000733 seconds)
gcworker stdout | 2025-02-07 04:44:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:54 UTC)" (scheduled at 2025-02-07 04:44:24.423055+00:00)
gcworker stdout | 2025-02-07 04:44:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:44:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:44:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:44:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:44:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:38.732147+00:00 (in 9.999503 seconds)
notificationworker stdout | 2025-02-07 04:44:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:38 UTC)" (scheduled at 2025-02-07 04:44:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:44:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:44:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 28, 732960), True, datetime.datetime(2025, 2, 7, 4, 44, 28, 732960), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:44:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:44:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:44:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:44:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:44:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:44.225378+00:00 (in 14.999534 seconds)
quotatotalworker stdout | 2025-02-07 04:44:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:44 UTC)" (scheduled at 2025-02-07 04:44:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:44:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:44:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:44:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:44:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:44:29,235 [235] [DEBUG] [app] Starting request: urn:request:39d06ebf-ff88-4ae2-a7ae-695e84852c1f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:29,235 [236] [DEBUG] [app] Starting request: urn:request:93532f3b-746a-4e23-9710-0d9b685c0aac (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:44:29,239 [239] [DEBUG] [app] Starting request: urn:request:c63f3a14-021c-4b1f-85ad-8b666827c65a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:29,239 [245] [DEBUG] [app] Starting request: urn:request:ab30c678-eec6-43af-90a4-11372bc2fe84 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:29,239 [239] [DEBUG] [app] Ending request: urn:request:c63f3a14-021c-4b1f-85ad-8b666827c65a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c63f3a14-021c-4b1f-85ad-8b666827c65a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:44:29,239 [245] [DEBUG] [app] Ending request: urn:request:ab30c678-eec6-43af-90a4-11372bc2fe84 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab30c678-eec6-43af-90a4-11372bc2fe84', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:44:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:44:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:44:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:29,242 [236] [DEBUG] [app] Starting request: urn:request:770b20ce-32dd-413b-ab6a-24dd2a8ca21b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:29,243 [236] [DEBUG] [app] Ending request: urn:request:770b20ce-32dd-413b-ab6a-24dd2a8ca21b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:770b20ce-32dd-413b-ab6a-24dd2a8ca21b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:44:29,243 [238] [DEBUG] [app] Starting request: urn:request:fd3ef7eb-4b36-45dd-b35f-363da19542ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:44:29,243 [238] [DEBUG] [app] Ending request: urn:request:fd3ef7eb-4b36-45dd-b35f-363da19542ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd3ef7eb-4b36-45dd-b35f-363da19542ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:44:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:44:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:29,255 [235] [DEBUG] [app] Ending request: urn:request:39d06ebf-ff88-4ae2-a7ae-695e84852c1f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39d06ebf-ff88-4ae2-a7ae-695e84852c1f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:44:29,255 [236] [DEBUG] [app] Ending request: urn:request:93532f3b-746a-4e23-9710-0d9b685c0aac (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:93532f3b-746a-4e23-9710-0d9b685c0aac', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:44:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:44:29,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
storagereplication stdout | 2025-02-07 04:44:29,386 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:44:31,115 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:44:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:44:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:32.625945+00:00 (in 1.004310 seconds)
namespacegcworker stdout | 2025-02-07 04:44:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:31 UTC)" (scheduled at 2025-02-07 04:44:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:44:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:44:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 31, 621947), True, datetime.datetime(2025, 2, 7, 4, 44, 31, 621947), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:44:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:44:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:44:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:44:32,287 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:44:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:44:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:31.621131+00:00 (in 58.994634 seconds)
namespacegcworker stdout | 2025-02-07 04:44:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:32 UTC)" (scheduled at 2025-02-07 04:44:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:44:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:44:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:44:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:44:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:03.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 04:44:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:03 UTC)" (scheduled at 2025-02-07 04:44:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:44:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 44, 33, 919994), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:44:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:44:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:44:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:44:36,602 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:44:37,642 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:56.713547+00:00 (in 18.997220 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:37 UTC)" (scheduled at 2025-02-07 04:44:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:44:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:44:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:48.732147+00:00 (in 9.999477 seconds)
notificationworker stdout | 2025-02-07 04:44:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:48 UTC)" (scheduled at 2025-02-07 04:44:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:44:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:44:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 38, 732967), True, datetime.datetime(2025, 2, 7, 4, 44, 38, 732967), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:44:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:44:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:44:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:44:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:44:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:54.423055+00:00 (in 13.998376 seconds)
gcworker stdout | 2025-02-07 04:44:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:10 UTC)" (scheduled at 2025-02-07 04:44:40.424250+00:00)
gcworker stdout | 2025-02-07 04:44:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:44:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903180431, None, 1, 0])
gcworker stdout | 2025-02-07 04:44:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:44:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:44:41,594 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:44:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:44:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:11.831521+00:00 (in 29.999522 seconds)
autopruneworker stdout | 2025-02-07 04:44:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:11 UTC)" (scheduled at 2025-02-07 04:44:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:44:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899881836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:44:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:44:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:44:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:44:42,235 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:44:42,661 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:44:44,022 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:44:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:44:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:59.225378+00:00 (in 14.999523 seconds)
quotatotalworker stdout | 2025-02-07 04:44:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:59 UTC)" (scheduled at 2025-02-07 04:44:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:44:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:44:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:44:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:44:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:44:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:44:44,235 [235] [DEBUG] [app] Starting request: urn:request:708c64c8-a51b-4793-93a2-8f6972875acf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:44,235 [236] [DEBUG] [app] Starting request: urn:request:c36ff817-415a-437f-bfa8-ab9c18d76428 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:44:44,238 [243] [DEBUG] [app] Starting request: urn:request:dc8ab0f5-062c-4c5f-9257-86639095157f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:44,239 [239] [DEBUG] [app] Starting request: urn:request:c881bc9a-b1f8-4d7c-9b64-e581fe3dd769 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:44,239 [243] [DEBUG] [app] Ending request: urn:request:dc8ab0f5-062c-4c5f-9257-86639095157f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc8ab0f5-062c-4c5f-9257-86639095157f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:44:44,239 [239] [DEBUG] [app] Ending request: urn:request:c881bc9a-b1f8-4d7c-9b64-e581fe3dd769 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c881bc9a-b1f8-4d7c-9b64-e581fe3dd769', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:44:44,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:44:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:44:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:44,242 [236] [DEBUG] [app] Starting request: urn:request:dd7aafa7-00af-49c0-a911-a0eca654d6eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:44,242 [236] [DEBUG] [app] Ending request: urn:request:dd7aafa7-00af-49c0-a911-a0eca654d6eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd7aafa7-00af-49c0-a911-a0eca654d6eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:44:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:44:44,243 [238] [DEBUG] [app] Starting request: urn:request:7323b1bf-a419-4c96-b073-d2de026e0753 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:44,243 [238] [DEBUG] [app] Ending request: urn:request:7323b1bf-a419-4c96-b073-d2de026e0753 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7323b1bf-a419-4c96-b073-d2de026e0753', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:44:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:44:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:44:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:44,254 [236] [DEBUG] [app] Ending request: urn:request:c36ff817-415a-437f-bfa8-ab9c18d76428 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c36ff817-415a-437f-bfa8-ab9c18d76428', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:44:44,254 [235] [DEBUG] [app] Ending request: urn:request:708c64c8-a51b-4793-93a2-8f6972875acf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:708c64c8-a51b-4793-93a2-8f6972875acf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:44:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 04:44:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
notificationworker stdout | 2025-02-07 04:44:44,911 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:44:45,014 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:44:46,074 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:46,555 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:44:47,009 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:44:47,013 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:44:47,015 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:44:47,019 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:44:47,022 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:44:47,078 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:44:47,089 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:44:47,207 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:44:47,665 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:44:48,036 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:44:48,039 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:44:48,043 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:44:48,122 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:44:48,285 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:44:48,379 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:44:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:44:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:50.814677+00:00 (in 2.082065 seconds)
notificationworker stdout | 2025-02-07 04:44:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:58 UTC)" (scheduled at 2025-02-07 04:44:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:44:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:44:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 48, 732886), True, datetime.datetime(2025, 2, 7, 4, 44, 48, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:44:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:44:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:44:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:44:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:44:49,273 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:44:49,276 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:44:49,279 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:44:49,284 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:44:49,286 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:44:49,289 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:44:49,293 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:44:49,295 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:44:49,298 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:44:49,937 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:44:49,974 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:44:50,403 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:44:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:44:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:58.732147+00:00 (in 7.917029 seconds)
notificationworker stdout | 2025-02-07 04:44:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:50 UTC)" (scheduled at 2025-02-07 04:44:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:44:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:44:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:44:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:44:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:44:59.937899+00:00 (in 6.999794 seconds)
exportactionlogsworker stdout | 2025-02-07 04:44:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:49:52 UTC)" (scheduled at 2025-02-07 04:44:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:44:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 44, 52, 938421), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:44:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 44, 52, 938421), True, datetime.datetime(2025, 2, 7, 4, 44, 52, 938421), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:44:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 44, 52, 938421), True, datetime.datetime(2025, 2, 7, 4, 44, 52, 938421), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 44, 52, 938421), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:44:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:44:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:49:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:44:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:44:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:10.424250+00:00 (in 16.000736 seconds)
gcworker stdout | 2025-02-07 04:44:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:24 UTC)" (scheduled at 2025-02-07 04:44:54.423055+00:00)
gcworker stdout | 2025-02-07 04:44:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:44:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:44:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:44:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:56.118855+00:00 (in 59.999510 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:44:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:56 UTC)" (scheduled at 2025-02-07 04:44:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:44:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:44:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:44:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:37.715878+00:00 (in 41.001875 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:56 UTC)" (scheduled at 2025-02-07 04:44:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:44:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:44:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:44:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:08.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 04:44:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:08 UTC)" (scheduled at 2025-02-07 04:44:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:44:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:44:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 44, 58, 732873), True, datetime.datetime(2025, 2, 7, 4, 44, 58, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:44:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:44:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:44:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:44:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:44:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:14.225378+00:00 (in 14.999512 seconds)
quotatotalworker stdout | 2025-02-07 04:44:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:14 UTC)" (scheduled at 2025-02-07 04:44:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:44:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:44:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:44:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:44:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:44:59,235 [237] [DEBUG] [app] Starting request: urn:request:b281810c-7ba6-4d10-ab29-895dbbed1537 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:59,235 [235] [DEBUG] [app] Starting request: urn:request:a308297a-0832-42dc-91bb-aada9cc5ea7e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:44:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:44:59,238 [246] [DEBUG] [app] Starting request: urn:request:da356afd-f167-4f2e-98e8-10f549852f40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:59,238 [239] [DEBUG] [app] Starting request: urn:request:0eaf95fa-53ed-4ef7-9593-8d57bbbf96cf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:44:59,239 [246] [DEBUG] [app] Ending request: urn:request:da356afd-f167-4f2e-98e8-10f549852f40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da356afd-f167-4f2e-98e8-10f549852f40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:44:59,239 [239] [DEBUG] [app] Ending request: urn:request:0eaf95fa-53ed-4ef7-9593-8d57bbbf96cf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0eaf95fa-53ed-4ef7-9593-8d57bbbf96cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:44:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:44:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:44:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:44:59,241 [237] [DEBUG] [app] Starting request: urn:request:394ab3ec-3678-4f8e-b066-26b52b3c69aa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:59,242 [237] [DEBUG] [app] Ending request: urn:request:394ab3ec-3678-4f8e-b066-26b52b3c69aa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:394ab3ec-3678-4f8e-b066-26b52b3c69aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:44:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:44:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:59,242 [238] [DEBUG] [app] Starting request: urn:request:87a6c4d8-1bb0-4df4-9ea0-ca0dabace783 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:44:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:59,243 [238] [DEBUG] [app] Ending request: urn:request:87a6c4d8-1bb0-4df4-9ea0-ca0dabace783 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87a6c4d8-1bb0-4df4-9ea0-ca0dabace783', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:44:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:44:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:44:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:44:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:44:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:44:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:44:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:44:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:44:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:44:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:59,253 [237] [DEBUG] [app] Ending request: urn:request:b281810c-7ba6-4d10-ab29-895dbbed1537 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b281810c-7ba6-4d10-ab29-895dbbed1537', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:44:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:44:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:44:59,254 [235] [DEBUG] [app] Ending request: urn:request:a308297a-0832-42dc-91bb-aada9cc5ea7e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a308297a-0832-42dc-91bb-aada9cc5ea7e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:44:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:44:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:44:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:44:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 04:44:59,418 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:44:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:44:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:04.936688+00:00 (in 4.998301 seconds)
exportactionlogsworker stdout | 2025-02-07 04:44:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:59 UTC)" (scheduled at 2025-02-07 04:44:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:44:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:44:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:45:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:45:01,141 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:45:02,324 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:45:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:45:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:23.420886+00:00 (in 20.001337 seconds)
repositorygcworker stdout | 2025-02-07 04:45:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:03 UTC)" (scheduled at 2025-02-07 04:45:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:45:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:45:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 3, 419864), True, datetime.datetime(2025, 2, 7, 4, 45, 3, 419864), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:45:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:45:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:45:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:45:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:45:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:33.919286+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2025-02-07 04:45:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:33 UTC)" (scheduled at 2025-02-07 04:45:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:45:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 45, 3, 920016), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:45:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:45:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:45:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:45:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:45:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:59.937899+00:00 (in 55.000763 seconds)
exportactionlogsworker stdout | 2025-02-07 04:45:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:04 UTC)" (scheduled at 2025-02-07 04:45:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:45:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:45:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 4, 937462), True, datetime.datetime(2025, 2, 7, 4, 45, 4, 937462), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:45:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:45:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:45:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:45:06,618 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:45:07,678 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:45:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:45:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:18.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 04:45:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:18 UTC)" (scheduled at 2025-02-07 04:45:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:45:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:45:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 8, 732888), True, datetime.datetime(2025, 2, 7, 4, 45, 8, 732888), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:45:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:45:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:45:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:45:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:45:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:24.423055+00:00 (in 13.998372 seconds)
gcworker stdout | 2025-02-07 04:45:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:40 UTC)" (scheduled at 2025-02-07 04:45:10.424250+00:00)
gcworker stdout | 2025-02-07 04:45:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:45:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903210431, None, 1, 0])
gcworker stdout | 2025-02-07 04:45:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:45:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:45:11,626 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:45:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:45:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:41.831521+00:00 (in 29.999570 seconds)
autopruneworker stdout | 2025-02-07 04:45:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:41 UTC)" (scheduled at 2025-02-07 04:45:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:45:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899911836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:45:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:45:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:45:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:45:12,245 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:45:12,690 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:45:14,058 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:45:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:45:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:29.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:45:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:29 UTC)" (scheduled at 2025-02-07 04:45:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:45:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:45:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:45:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:45:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:45:14,235 [238] [DEBUG] [app] Starting request: urn:request:d34f3cea-1511-44dc-b01b-796a6417e869 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:14,235 [235] [DEBUG] [app] Starting request: urn:request:0aeeb671-2a45-4035-a038-9aca5850ca06 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:45:14,238 [245] [DEBUG] [app] Starting request: urn:request:fbe203cd-a4d4-4bd2-a4e5-84f386e52773 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:14,238 [244] [DEBUG] [app] Starting request: urn:request:86050f65-8255-40c1-b03e-ae20f25b89b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:14,238 [245] [DEBUG] [app] Ending request: urn:request:fbe203cd-a4d4-4bd2-a4e5-84f386e52773 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fbe203cd-a4d4-4bd2-a4e5-84f386e52773', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:45:14,239 [244] [DEBUG] [app] Ending request: urn:request:86050f65-8255-40c1-b03e-ae20f25b89b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86050f65-8255-40c1-b03e-ae20f25b89b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:45:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:45:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:45:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:14,241 [238] [DEBUG] [app] Starting request: urn:request:80dde6f9-132c-45b9-978a-dde46f18a9b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:45:14,241 [236] [DEBUG] [app] Starting request: urn:request:d870e5f7-dcf1-45c3-8635-50b632090438 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:45:14,241 [238] [DEBUG] [app] Ending request: urn:request:80dde6f9-132c-45b9-978a-dde46f18a9b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:80dde6f9-132c-45b9-978a-dde46f18a9b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:45:14,242 [236] [DEBUG] [app] Ending request: urn:request:d870e5f7-dcf1-45c3-8635-50b632090438 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d870e5f7-dcf1-45c3-8635-50b632090438', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:45:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:45:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:45:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:45:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:45:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:14,253 [235] [DEBUG] [app] Ending request: urn:request:0aeeb671-2a45-4035-a038-9aca5850ca06 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0aeeb671-2a45-4035-a038-9aca5850ca06', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:45:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:45:14,253 [238] [DEBUG] [app] Ending request: urn:request:d34f3cea-1511-44dc-b01b-796a6417e869 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d34f3cea-1511-44dc-b01b-796a6417e869', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:45:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:45:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:45:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:45:14,948 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:45:15,022 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:45:16,110 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:16,591 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:45:17,018 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:45:17,021 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:45:17,024 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:45:17,027 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:45:17,030 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:45:17,113 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:45:17,124 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:45:17,230 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:45:17,701 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:45:18,044 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:45:18,047 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:45:18,049 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:45:18,158 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:45:18,314 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:45:18,414 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:45:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:45:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:28.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 04:45:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:28 UTC)" (scheduled at 2025-02-07 04:45:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:45:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:45:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 18, 732871), True, datetime.datetime(2025, 2, 7, 4, 45, 18, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:45:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:45:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:45:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:45:19,282 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:45:19,287 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:45:19,290 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:45:19,293 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:45:19,295 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:45:19,298 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:45:19,303 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:45:19,306 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:45:19,308 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:45:19,973 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:45:19,983 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:45:20,437 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:45:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:45:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:03.419110+00:00 (in 39.997692 seconds)
repositorygcworker stdout | 2025-02-07 04:45:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:23 UTC)" (scheduled at 2025-02-07 04:45:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:45:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:45:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:45:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:45:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:40.424250+00:00 (in 16.000691 seconds)
gcworker stdout | 2025-02-07 04:45:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:54 UTC)" (scheduled at 2025-02-07 04:45:24.423055+00:00)
gcworker stdout | 2025-02-07 04:45:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:45:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:45:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:45:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:45:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:38.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 04:45:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:38 UTC)" (scheduled at 2025-02-07 04:45:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:45:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:45:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 28, 732869), True, datetime.datetime(2025, 2, 7, 4, 45, 28, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:45:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:45:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:45:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:45:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:45:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:44.225378+00:00 (in 14.999561 seconds)
quotatotalworker stdout | 2025-02-07 04:45:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:44 UTC)" (scheduled at 2025-02-07 04:45:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:45:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:45:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:45:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:45:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:45:29,235 [235] [DEBUG] [app] Starting request: urn:request:dc733ef7-ab7a-4ce2-bbf3-b6f3d30a7b56 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:29,235 [238] [DEBUG] [app] Starting request: urn:request:c09eee3d-b596-4bf5-b4b8-71edb9d23635 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:45:29,238 [246] [DEBUG] [app] Starting request: urn:request:c56e605c-7642-48c1-88cc-31476764028b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:29,238 [239] [DEBUG] [app] Starting request: urn:request:f11c45b2-3c50-442b-959f-5843dacc4ec1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:29,238 [246] [DEBUG] [app] Ending request: urn:request:c56e605c-7642-48c1-88cc-31476764028b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c56e605c-7642-48c1-88cc-31476764028b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:45:29,239 [239] [DEBUG] [app] Ending request: urn:request:f11c45b2-3c50-442b-959f-5843dacc4ec1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f11c45b2-3c50-442b-959f-5843dacc4ec1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:45:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:45:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:29,241 [238] [DEBUG] [app] Starting request: urn:request:4c8ada40-012b-4f00-836c-896e96e58ca6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:45:29,242 [238] [DEBUG] [app] Ending request: urn:request:4c8ada40-012b-4f00-836c-896e96e58ca6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4c8ada40-012b-4f00-836c-896e96e58ca6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:45:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:29,242 [237] [DEBUG] [app] Starting request: urn:request:3c67af9b-24fa-4c7e-888a-6a02e021a90d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:45:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:45:29,243 [237] [DEBUG] [app] Ending request: urn:request:3c67af9b-24fa-4c7e-888a-6a02e021a90d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c67af9b-24fa-4c7e-888a-6a02e021a90d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:45:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:45:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:29,254 [235] [DEBUG] [app] Ending request: urn:request:dc733ef7-ab7a-4ce2-bbf3-b6f3d30a7b56 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc733ef7-ab7a-4ce2-bbf3-b6f3d30a7b56', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:45:29,254 [238] [DEBUG] [app] Ending request: urn:request:c09eee3d-b596-4bf5-b4b8-71edb9d23635 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c09eee3d-b596-4bf5-b4b8-71edb9d23635', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:45:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:45:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:45:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:45:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 04:45:29,435 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:45:31,177 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:45:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:45:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:32.625945+00:00 (in 1.004332 seconds)
namespacegcworker stdout | 2025-02-07 04:45:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:31 UTC)" (scheduled at 2025-02-07 04:45:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:45:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:45:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 31, 621883), True, datetime.datetime(2025, 2, 7, 4, 45, 31, 621883), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:45:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:45:31,628 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:45:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:45:32,343 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:45:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:45:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:31.621131+00:00 (in 58.994741 seconds)
namespacegcworker stdout | 2025-02-07 04:45:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:32 UTC)" (scheduled at 2025-02-07 04:45:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:45:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:45:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:45:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:45:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:03.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 04:45:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:03 UTC)" (scheduled at 2025-02-07 04:45:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:45:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 45, 33, 919988), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:45:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:45:33,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:45:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:45:36,654 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:45:37,714 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:56.713547+00:00 (in 18.997238 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:37 UTC)" (scheduled at 2025-02-07 04:45:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:45:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:45:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:48.732147+00:00 (in 9.999541 seconds)
notificationworker stdout | 2025-02-07 04:45:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:48 UTC)" (scheduled at 2025-02-07 04:45:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:45:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:45:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 38, 732887), True, datetime.datetime(2025, 2, 7, 4, 45, 38, 732887), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:45:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:45:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:45:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:45:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:45:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:54.423055+00:00 (in 13.998256 seconds)
gcworker stdout | 2025-02-07 04:45:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:10 UTC)" (scheduled at 2025-02-07 04:45:40.424250+00:00)
gcworker stdout | 2025-02-07 04:45:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:45:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903240431, None, 1, 0])
gcworker stdout | 2025-02-07 04:45:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:45:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:45:41,663 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:45:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:45:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:11.831521+00:00 (in 29.999562 seconds)
autopruneworker stdout | 2025-02-07 04:45:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:11 UTC)" (scheduled at 2025-02-07 04:45:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:45:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899941836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:45:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:45:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:45:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:45:42,274 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:45:42,718 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:45:44,084 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:45:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:45:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:59.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 04:45:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:59 UTC)" (scheduled at 2025-02-07 04:45:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:45:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:45:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:45:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:45:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:45:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:45:44,236 [238] [DEBUG] [app] Starting request: urn:request:d25e0356-5eff-45c4-aab6-6a7c150ca714 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:44,236 [235] [DEBUG] [app] Starting request: urn:request:702cd005-bc39-4695-a6bd-91cd4c014d59 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:45:44,239 [245] [DEBUG] [app] Starting request: urn:request:c5009004-6186-4e10-8bf9-b8534e3a8d9d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:44,239 [239] [DEBUG] [app] Starting request: urn:request:195965e2-ae29-4bc6-9958-c0d4e0caecf2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:44,240 [245] [DEBUG] [app] Ending request: urn:request:c5009004-6186-4e10-8bf9-b8534e3a8d9d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5009004-6186-4e10-8bf9-b8534e3a8d9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:45:44,240 [239] [DEBUG] [app] Ending request: urn:request:195965e2-ae29-4bc6-9958-c0d4e0caecf2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:195965e2-ae29-4bc6-9958-c0d4e0caecf2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:45:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:45:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:44,243 [237] [DEBUG] [app] Starting request: urn:request:0377750a-ca35-4066-aac9-8fbfb2539909 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:45:44,243 [237] [DEBUG] [app] Ending request: urn:request:0377750a-ca35-4066-aac9-8fbfb2539909 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0377750a-ca35-4066-aac9-8fbfb2539909', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:45:44,243 [236] [DEBUG] [app] Starting request: urn:request:b070482b-92ce-4942-9f83-cdc3b8a8d438 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:45:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:44,244 [236] [DEBUG] [app] Ending request: urn:request:b070482b-92ce-4942-9f83-cdc3b8a8d438 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b070482b-92ce-4942-9f83-cdc3b8a8d438', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:45:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:45:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:45:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:44,254 [238] [DEBUG] [app] Ending request: urn:request:d25e0356-5eff-45c4-aab6-6a7c150ca714 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d25e0356-5eff-45c4-aab6-6a7c150ca714', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:45:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:45:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:45:44,255 [235] [DEBUG] [app] Ending request: urn:request:702cd005-bc39-4695-a6bd-91cd4c014d59 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:702cd005-bc39-4695-a6bd-91cd4c014d59', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:45:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:45:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:45:44,968 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:45:45,036 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:45:46,146 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:46,613 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:45:47,026 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:45:47,030 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:45:47,033 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:45:47,037 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:45:47,040 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:45:47,139 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:45:47,159 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:45:47,266 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:45:47,722 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:45:48,052 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:45:48,055 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:45:48,059 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:45:48,168 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:45:48,340 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:45:48,448 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:45:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:45:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:50.814677+00:00 (in 2.081943 seconds)
notificationworker stdout | 2025-02-07 04:45:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:58 UTC)" (scheduled at 2025-02-07 04:45:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:45:48,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:45:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 48, 733128), True, datetime.datetime(2025, 2, 7, 4, 45, 48, 733128), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:45:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:45:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:45:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:45:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:45:49,291 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:45:49,299 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:45:49,302 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:45:49,306 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:45:49,308 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:45:49,311 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:45:49,314 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:45:49,317 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:45:49,320 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:45:50,009 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:45:50,018 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:45:50,446 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:45:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:45:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:45:58.732147+00:00 (in 7.916993 seconds)
notificationworker stdout | 2025-02-07 04:45:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:50 UTC)" (scheduled at 2025-02-07 04:45:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:45:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:45:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:45:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:45:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:10.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 04:45:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:24 UTC)" (scheduled at 2025-02-07 04:45:54.423055+00:00)
gcworker stdout | 2025-02-07 04:45:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:45:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:45:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:45:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:56.118855+00:00 (in 59.999505 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:45:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:56 UTC)" (scheduled at 2025-02-07 04:45:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:45:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:45:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:45:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:37.715878+00:00 (in 41.001870 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:56 UTC)" (scheduled at 2025-02-07 04:45:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:45:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:45:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:45:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:08.732147+00:00 (in 9.999396 seconds)
notificationworker stdout | 2025-02-07 04:45:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:08 UTC)" (scheduled at 2025-02-07 04:45:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:45:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:45:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 45, 58, 733034), True, datetime.datetime(2025, 2, 7, 4, 45, 58, 733034), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:45:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:45:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:45:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:45:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:45:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:14.225378+00:00 (in 14.999523 seconds)
quotatotalworker stdout | 2025-02-07 04:45:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:14 UTC)" (scheduled at 2025-02-07 04:45:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:45:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:45:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:45:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:45:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:45:59,235 [235] [DEBUG] [app] Starting request: urn:request:411a8cb2-fe55-486e-8df6-3e6f4c5ce9d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:59,235 [238] [DEBUG] [app] Starting request: urn:request:273775ee-57ca-406d-b0c8-4740e41e3105 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:45:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:45:59,238 [244] [DEBUG] [app] Starting request: urn:request:03fc2655-f7ba-468d-a1d1-05e0d0c967cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:59,239 [239] [DEBUG] [app] Starting request: urn:request:ee8ef59c-4bba-49b2-bc07-3fe9cec5f7a6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:45:59,239 [244] [DEBUG] [app] Ending request: urn:request:03fc2655-f7ba-468d-a1d1-05e0d0c967cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03fc2655-f7ba-468d-a1d1-05e0d0c967cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:45:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:45:59,239 [239] [DEBUG] [app] Ending request: urn:request:ee8ef59c-4bba-49b2-bc07-3fe9cec5f7a6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee8ef59c-4bba-49b2-bc07-3fe9cec5f7a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:45:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:45:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:45:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:45:59,242 [235] [DEBUG] [app] Starting request: urn:request:8173b198-26eb-42cd-af3d-259af974921b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:45:59,243 [235] [DEBUG] [app] Ending request: urn:request:8173b198-26eb-42cd-af3d-259af974921b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8173b198-26eb-42cd-af3d-259af974921b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:45:59,243 [237] [DEBUG] [app] Starting request: urn:request:385583f0-04ca-4485-8142-878460598b56 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:45:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:45:59,243 [237] [DEBUG] [app] Ending request: urn:request:385583f0-04ca-4485-8142-878460598b56 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:385583f0-04ca-4485-8142-878460598b56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:45:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:45:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:45:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:45:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:45:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:45:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:45:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:45:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:45:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:45:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:45:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:45:59,255 [235] [DEBUG] [app] Ending request: urn:request:411a8cb2-fe55-486e-8df6-3e6f4c5ce9d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:411a8cb2-fe55-486e-8df6-3e6f4c5ce9d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:45:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:45:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:45:59,256 [238] [DEBUG] [app] Ending request: urn:request:273775ee-57ca-406d-b0c8-4740e41e3105 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:273775ee-57ca-406d-b0c8-4740e41e3105', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:45:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:45:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:45:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:45:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
storagereplication stdout | 2025-02-07 04:45:59,471 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:45:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:45:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:04.936688+00:00 (in 4.998306 seconds)
exportactionlogsworker stdout | 2025-02-07 04:45:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:59 UTC)" (scheduled at 2025-02-07 04:45:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:45:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:45:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:46:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:46:01,211 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:46:02,362 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:46:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:46:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:23.420886+00:00 (in 20.001282 seconds)
repositorygcworker stdout | 2025-02-07 04:46:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:03 UTC)" (scheduled at 2025-02-07 04:46:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:46:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:46:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 3, 419874), True, datetime.datetime(2025, 2, 7, 4, 46, 3, 419874), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:46:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:46:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:46:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:46:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:46:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:33.919286+00:00 (in 29.999554 seconds)
buildlogsarchiver stdout | 2025-02-07 04:46:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:33 UTC)" (scheduled at 2025-02-07 04:46:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:46:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 46, 3, 919977), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:46:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:46:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:46:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:46:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:46:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:59.937899+00:00 (in 55.000749 seconds)
exportactionlogsworker stdout | 2025-02-07 04:46:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:04 UTC)" (scheduled at 2025-02-07 04:46:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:46:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:46:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 4, 937463), True, datetime.datetime(2025, 2, 7, 4, 46, 4, 937463), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:46:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:46:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:46:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:46:06,669 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:46:07,742 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:46:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:46:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:18.732147+00:00 (in 9.999611 seconds)
notificationworker stdout | 2025-02-07 04:46:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:18 UTC)" (scheduled at 2025-02-07 04:46:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:46:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:46:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 8, 732823), True, datetime.datetime(2025, 2, 7, 4, 46, 8, 732823), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:46:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:46:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:46:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:46:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:46:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:24.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 04:46:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:40 UTC)" (scheduled at 2025-02-07 04:46:10.424250+00:00)
gcworker stdout | 2025-02-07 04:46:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:46:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903270430, None, 1, 0])
gcworker stdout | 2025-02-07 04:46:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:46:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:46:11,700 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:46:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:46:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:41.831521+00:00 (in 29.999507 seconds)
autopruneworker stdout | 2025-02-07 04:46:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:41 UTC)" (scheduled at 2025-02-07 04:46:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:46:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738899971836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:46:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:46:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:46:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:46:12,281 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:46:12,731 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:46:14,118 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:46:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:46:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:29.225378+00:00 (in 14.999494 seconds)
quotatotalworker stdout | 2025-02-07 04:46:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:29 UTC)" (scheduled at 2025-02-07 04:46:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:46:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:46:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:46:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:46:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:46:14,235 [235] [DEBUG] [app] Starting request: urn:request:6eeab987-4aee-4ad0-a1d3-f85e9245043d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:14,235 [238] [DEBUG] [app] Starting request: urn:request:38216131-520c-4efd-a444-b938eb5d4cfc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:46:14,237 [239] [DEBUG] [app] Starting request: urn:request:0ba376e1-cdc5-4394-888e-64993d6d7a56 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:14,237 [244] [DEBUG] [app] Starting request: urn:request:f4623b1c-1491-4650-923c-b839563d3094 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:14,238 [239] [DEBUG] [app] Ending request: urn:request:0ba376e1-cdc5-4394-888e-64993d6d7a56 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ba376e1-cdc5-4394-888e-64993d6d7a56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:46:14,238 [244] [DEBUG] [app] Ending request: urn:request:f4623b1c-1491-4650-923c-b839563d3094 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4623b1c-1491-4650-923c-b839563d3094', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:46:14,238 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:46:14,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:46:14,238 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:14,238 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:14,242 [237] [DEBUG] [app] Starting request: urn:request:7317b19f-a719-408a-adf3-1133aefdfefc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:14,242 [236] [DEBUG] [app] Starting request: urn:request:1350019e-9a4a-45bc-9a28-66e6847c0aa9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:14,242 [236] [DEBUG] [app] Ending request: urn:request:1350019e-9a4a-45bc-9a28-66e6847c0aa9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1350019e-9a4a-45bc-9a28-66e6847c0aa9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:46:14,242 [237] [DEBUG] [app] Ending request: urn:request:7317b19f-a719-408a-adf3-1133aefdfefc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7317b19f-a719-408a-adf3-1133aefdfefc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:46:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:46:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:46:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:14,254 [238] [DEBUG] [app] Ending request: urn:request:38216131-520c-4efd-a444-b938eb5d4cfc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38216131-520c-4efd-a444-b938eb5d4cfc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:14,254 [235] [DEBUG] [app] Ending request: urn:request:6eeab987-4aee-4ad0-a1d3-f85e9245043d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6eeab987-4aee-4ad0-a1d3-f85e9245043d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:46:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:46:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:46:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:46:15,003 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:46:15,072 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:46:16,183 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:16,623 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:46:17,035 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:46:17,039 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:46:17,041 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:46:17,044 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:46:17,047 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:46:17,177 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:46:17,190 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:46:17,302 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:46:17,758 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:46:18,060 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:46:18,064 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:46:18,066 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:46:18,204 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:46:18,376 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:46:18,474 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:46:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:46:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:28.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 04:46:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:28 UTC)" (scheduled at 2025-02-07 04:46:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:46:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:46:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 18, 732882), True, datetime.datetime(2025, 2, 7, 4, 46, 18, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:46:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:46:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:46:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:46:19,303 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:46:19,310 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:46:19,313 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:46:19,317 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:46:19,320 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:46:19,324 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:46:19,328 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:46:19,330 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:46:19,333 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
buildlogsarchiver stdout | 2025-02-07 04:46:20,034 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:46:20,037 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:46:20,482 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:46:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:46:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:03.419110+00:00 (in 39.997715 seconds)
repositorygcworker stdout | 2025-02-07 04:46:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:23 UTC)" (scheduled at 2025-02-07 04:46:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:46:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:46:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:46:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:46:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:40.424250+00:00 (in 16.000665 seconds)
gcworker stdout | 2025-02-07 04:46:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:54 UTC)" (scheduled at 2025-02-07 04:46:24.423055+00:00)
gcworker stdout | 2025-02-07 04:46:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:46:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:46:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:46:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:46:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:38.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 04:46:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:38 UTC)" (scheduled at 2025-02-07 04:46:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:46:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:46:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 28, 732862), True, datetime.datetime(2025, 2, 7, 4, 46, 28, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:46:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:46:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:46:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:46:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:46:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:44.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 04:46:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:44 UTC)" (scheduled at 2025-02-07 04:46:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:46:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:46:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:46:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:46:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:46:29,236 [238] [DEBUG] [app] Starting request: urn:request:bdceddb5-d180-474b-94b7-3d2add4658db (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:29,236 [235] [DEBUG] [app] Starting request: urn:request:cd41224a-10d5-4fb9-b456-e375a263f673 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:46:29,238 [246] [DEBUG] [app] Starting request: urn:request:7eaaa76f-072b-4040-91de-1636a3ec9aed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:29,239 [245] [DEBUG] [app] Starting request: urn:request:989560f5-9c98-4554-b505-7049d9a45d07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:29,239 [246] [DEBUG] [app] Ending request: urn:request:7eaaa76f-072b-4040-91de-1636a3ec9aed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7eaaa76f-072b-4040-91de-1636a3ec9aed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:46:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:46:29,239 [245] [DEBUG] [app] Ending request: urn:request:989560f5-9c98-4554-b505-7049d9a45d07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:989560f5-9c98-4554-b505-7049d9a45d07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:46:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:46:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:46:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:29,242 [238] [DEBUG] [app] Starting request: urn:request:1a226d41-6253-42ea-872d-e01aeab1c89f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:29,242 [238] [DEBUG] [app] Ending request: urn:request:1a226d41-6253-42ea-872d-e01aeab1c89f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1a226d41-6253-42ea-872d-e01aeab1c89f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:46:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:46:29,242 [237] [DEBUG] [app] Starting request: urn:request:295aee54-064f-4d8c-a138-311d564e7387 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:29,243 [237] [DEBUG] [app] Ending request: urn:request:295aee54-064f-4d8c-a138-311d564e7387 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:295aee54-064f-4d8c-a138-311d564e7387', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:46:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:46:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:46:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:29,254 [235] [DEBUG] [app] Ending request: urn:request:cd41224a-10d5-4fb9-b456-e375a263f673 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd41224a-10d5-4fb9-b456-e375a263f673', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:29,254 [238] [DEBUG] [app] Ending request: urn:request:bdceddb5-d180-474b-94b7-3d2add4658db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bdceddb5-d180-474b-94b7-3d2add4658db', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:46:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:46:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:46:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 04:46:29,498 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:46:31,231 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:46:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:46:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:32.625945+00:00 (in 1.004344 seconds)
namespacegcworker stdout | 2025-02-07 04:46:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:31 UTC)" (scheduled at 2025-02-07 04:46:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:46:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:46:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 31, 621922), True, datetime.datetime(2025, 2, 7, 4, 46, 31, 621922), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:46:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:46:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:46:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:46:32,384 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:46:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:46:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:13.624301+00:00 (in 40.997906 seconds)
namespacegcworker stdout | 2025-02-07 04:46:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:32 UTC)" (scheduled at 2025-02-07 04:46:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:46:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:46:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:46:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:46:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:03.919286+00:00 (in 29.999542 seconds)
buildlogsarchiver stdout | 2025-02-07 04:46:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:03 UTC)" (scheduled at 2025-02-07 04:46:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:46:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 46, 33, 920014), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:46:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:46:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:46:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:46:36,705 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:56.713547+00:00 (in 18.997220 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:37 UTC)" (scheduled at 2025-02-07 04:46:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:46:37,778 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:46:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:46:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:48.732147+00:00 (in 9.999427 seconds)
notificationworker stdout | 2025-02-07 04:46:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:48 UTC)" (scheduled at 2025-02-07 04:46:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:46:38,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:46:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 38, 733095), True, datetime.datetime(2025, 2, 7, 4, 46, 38, 733095), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:46:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:46:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:46:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:46:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:46:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:54.423055+00:00 (in 13.998313 seconds)
gcworker stdout | 2025-02-07 04:46:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:10 UTC)" (scheduled at 2025-02-07 04:46:40.424250+00:00)
gcworker stdout | 2025-02-07 04:46:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:46:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903300431, None, 1, 0])
gcworker stdout | 2025-02-07 04:46:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:46:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:46:41,737 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:46:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:46:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:11.831521+00:00 (in 29.999486 seconds)
autopruneworker stdout | 2025-02-07 04:46:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:11 UTC)" (scheduled at 2025-02-07 04:46:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:46:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900001836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:46:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:46:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:46:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:46:42,298 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:46:42,767 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 04:46:44,154 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:46:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:46:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:59.225378+00:00 (in 14.999523 seconds)
quotatotalworker stdout | 2025-02-07 04:46:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:59 UTC)" (scheduled at 2025-02-07 04:46:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:46:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:46:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:46:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:46:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:46:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:46:44,235 [237] [DEBUG] [app] Starting request: urn:request:cd561e90-7603-4231-be2a-5610d964dc78 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:44,235 [238] [DEBUG] [app] Starting request: urn:request:ca8367d9-122d-44b8-a7b7-40aa330151cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:46:44,238 [239] [DEBUG] [app] Starting request: urn:request:05242a16-2a37-4328-a278-b5395a55aaeb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:44,238 [245] [DEBUG] [app] Starting request: urn:request:5cf9964c-af9c-4ecf-a8c5-2443d9951fba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:44,238 [239] [DEBUG] [app] Ending request: urn:request:05242a16-2a37-4328-a278-b5395a55aaeb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:05242a16-2a37-4328-a278-b5395a55aaeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:46:44,238 [245] [DEBUG] [app] Ending request: urn:request:5cf9964c-af9c-4ecf-a8c5-2443d9951fba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cf9964c-af9c-4ecf-a8c5-2443d9951fba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:46:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:46:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:46:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:44,241 [238] [DEBUG] [app] Starting request: urn:request:0779c2bf-b76b-4c25-892c-a11f6a0487d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:44,241 [238] [DEBUG] [app] Ending request: urn:request:0779c2bf-b76b-4c25-892c-a11f6a0487d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0779c2bf-b76b-4c25-892c-a11f6a0487d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:46:44,241 [236] [DEBUG] [app] Starting request: urn:request:5b461d93-5169-4a7c-8d62-930794bbd216 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:46:44,242 [236] [DEBUG] [app] Ending request: urn:request:5b461d93-5169-4a7c-8d62-930794bbd216 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5b461d93-5169-4a7c-8d62-930794bbd216', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:46:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:46:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:46:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:44,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:44,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:44,252 [237] [DEBUG] [app] Ending request: urn:request:cd561e90-7603-4231-be2a-5610d964dc78 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd561e90-7603-4231-be2a-5610d964dc78', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:46:44,253 [238] [DEBUG] [app] Ending request: urn:request:ca8367d9-122d-44b8-a7b7-40aa330151cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca8367d9-122d-44b8-a7b7-40aa330151cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:46:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 04:46:44,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:46:45,036 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:46:45,079 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:46:46,211 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:46,641 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:46:47,044 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:46:47,047 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:46:47,050 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:46:47,053 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:46:47,056 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:46:47,202 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:46:47,216 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:46:47,330 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:46:47,794 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:46:48,068 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:46:48,072 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:46:48,074 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:46:48,218 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:46:48,401 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:46:48,505 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:46:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:46:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:50.814677+00:00 (in 2.082074 seconds)
notificationworker stdout | 2025-02-07 04:46:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:58 UTC)" (scheduled at 2025-02-07 04:46:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:46:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:46:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 48, 732876), True, datetime.datetime(2025, 2, 7, 4, 46, 48, 732876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:46:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:46:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:46:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:46:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:46:49,316 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:46:49,318 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:46:49,321 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:46:49,325 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:46:49,329 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:46:49,332 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:46:49,337 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:46:49,340 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:46:49,342 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
buildlogsarchiver stdout | 2025-02-07 04:46:50,048 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:46:50,072 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:46:50,514 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:46:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:46:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:46:58.732147+00:00 (in 7.917003 seconds)
notificationworker stdout | 2025-02-07 04:46:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:50 UTC)" (scheduled at 2025-02-07 04:46:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:46:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:46:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:46:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:46:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:10.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 04:46:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:24 UTC)" (scheduled at 2025-02-07 04:46:54.423055+00:00)
gcworker stdout | 2025-02-07 04:46:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:46:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:46:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:46:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:56.118855+00:00 (in 59.999531 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:46:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:56 UTC)" (scheduled at 2025-02-07 04:46:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:46:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:46:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:46:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:37.715878+00:00 (in 41.001869 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:56 UTC)" (scheduled at 2025-02-07 04:46:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:46:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:46:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:46:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:08.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 04:46:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:08 UTC)" (scheduled at 2025-02-07 04:46:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:46:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:46:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 46, 58, 732865), True, datetime.datetime(2025, 2, 7, 4, 46, 58, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:46:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:46:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:46:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:46:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:46:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:14.225378+00:00 (in 14.999503 seconds)
quotatotalworker stdout | 2025-02-07 04:46:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:14 UTC)" (scheduled at 2025-02-07 04:46:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:46:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:46:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:46:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:46:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:46:59,235 [235] [DEBUG] [app] Starting request: urn:request:3acaafb2-5e90-4f4e-9043-1eefed4dbdc0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:59,235 [237] [DEBUG] [app] Starting request: urn:request:27e18444-bf54-4a1b-97b2-cc266ef48261 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:46:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:46:59,238 [239] [DEBUG] [app] Starting request: urn:request:9fe78edd-452c-4a6c-98db-85da7d6b6e14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:59,238 [246] [DEBUG] [app] Starting request: urn:request:7133b390-e69d-4185-aac2-6f21b9fb11e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:46:59,238 [239] [DEBUG] [app] Ending request: urn:request:9fe78edd-452c-4a6c-98db-85da7d6b6e14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9fe78edd-452c-4a6c-98db-85da7d6b6e14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:46:59,238 [246] [DEBUG] [app] Ending request: urn:request:7133b390-e69d-4185-aac2-6f21b9fb11e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7133b390-e69d-4185-aac2-6f21b9fb11e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:46:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:46:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:46:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:46:59,241 [237] [DEBUG] [app] Starting request: urn:request:56c94f52-e9c5-4d94-bbe6-fc5a4b62a06c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:59,241 [238] [DEBUG] [app] Starting request: urn:request:8ebeb3fc-067a-48ae-af02-fd6b6a836704 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:46:59,241 [237] [DEBUG] [app] Ending request: urn:request:56c94f52-e9c5-4d94-bbe6-fc5a4b62a06c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56c94f52-e9c5-4d94-bbe6-fc5a4b62a06c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:46:59,241 [238] [DEBUG] [app] Ending request: urn:request:8ebeb3fc-067a-48ae-af02-fd6b6a836704 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8ebeb3fc-067a-48ae-af02-fd6b6a836704', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:46:59,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:46:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:46:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:46:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:46:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:46:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:46:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:46:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:46:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:46:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:46:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:46:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:46:59,253 [237] [DEBUG] [app] Ending request: urn:request:27e18444-bf54-4a1b-97b2-cc266ef48261 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:27e18444-bf54-4a1b-97b2-cc266ef48261', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:59,253 [235] [DEBUG] [app] Ending request: urn:request:3acaafb2-5e90-4f4e-9043-1eefed4dbdc0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3acaafb2-5e90-4f4e-9043-1eefed4dbdc0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:46:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:46:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:46:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:46:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:46:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:46:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
storagereplication stdout | 2025-02-07 04:46:59,534 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:46:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:46:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:04.936688+00:00 (in 4.998326 seconds)
exportactionlogsworker stdout | 2025-02-07 04:46:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:59 UTC)" (scheduled at 2025-02-07 04:46:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:46:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:46:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:47:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:47:01,267 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:47:02,420 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:47:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:47:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:23.420886+00:00 (in 20.001276 seconds)
repositorygcworker stdout | 2025-02-07 04:47:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:03 UTC)" (scheduled at 2025-02-07 04:47:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:47:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:47:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 3, 419881), True, datetime.datetime(2025, 2, 7, 4, 47, 3, 419881), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:47:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:47:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:47:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:47:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:47:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:33.919286+00:00 (in 29.999473 seconds)
buildlogsarchiver stdout | 2025-02-07 04:47:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:33 UTC)" (scheduled at 2025-02-07 04:47:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:47:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 47, 3, 920100), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:47:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:47:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:47:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:47:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:47:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:59.937899+00:00 (in 55.000786 seconds)
exportactionlogsworker stdout | 2025-02-07 04:47:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:04 UTC)" (scheduled at 2025-02-07 04:47:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:47:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:47:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 4, 937401), True, datetime.datetime(2025, 2, 7, 4, 47, 4, 937401), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:47:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:47:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:47:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:47:06,742 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:47:07,815 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:47:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:18.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 04:47:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:18 UTC)" (scheduled at 2025-02-07 04:47:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:47:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:47:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 8, 732869), True, datetime.datetime(2025, 2, 7, 4, 47, 8, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:47:08,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:47:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:47:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:47:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:47:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:24.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 04:47:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:40 UTC)" (scheduled at 2025-02-07 04:47:10.424250+00:00)
gcworker stdout | 2025-02-07 04:47:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:47:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903330431, None, 1, 0])
gcworker stdout | 2025-02-07 04:47:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:47:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:47:11,773 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:47:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:47:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:41.831521+00:00 (in 29.999481 seconds)
autopruneworker stdout | 2025-02-07 04:47:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:41 UTC)" (scheduled at 2025-02-07 04:47:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:47:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900031836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:47:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:47:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:47:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:47:12,334 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:47:12,784 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
namespacegcworker stdout | 2025-02-07 04:47:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:47:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:31.621131+00:00 (in 17.996382 seconds)
namespacegcworker stdout | 2025-02-07 04:47:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:52:13 UTC)" (scheduled at 2025-02-07 04:47:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:47:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 47, 13, 625030), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:47:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 47, 13, 625030), True, datetime.datetime(2025, 2, 7, 4, 47, 13, 625030), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:47:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 47, 13, 625030), True, datetime.datetime(2025, 2, 7, 4, 47, 13, 625030), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 47, 13, 625030), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:47:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:47:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:52:13 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2025-02-07 04:47:14,190 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:47:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:47:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:29.225378+00:00 (in 14.999488 seconds)
quotatotalworker stdout | 2025-02-07 04:47:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:29 UTC)" (scheduled at 2025-02-07 04:47:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:47:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:47:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:47:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:47:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:47:14,236 [238] [DEBUG] [app] Starting request: urn:request:c9f3784e-b0fe-4538-a8d0-c7d6d36acda7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:14,236 [235] [DEBUG] [app] Starting request: urn:request:cabd21bf-8c11-483c-9976-5205f17966e6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:47:14,239 [246] [DEBUG] [app] Starting request: urn:request:11b198d1-1bea-45da-a6e2-5b500d620ee1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:14,239 [239] [DEBUG] [app] Starting request: urn:request:bd6686fb-677b-4cca-8151-a28b77ff21bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:14,240 [246] [DEBUG] [app] Ending request: urn:request:11b198d1-1bea-45da-a6e2-5b500d620ee1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11b198d1-1bea-45da-a6e2-5b500d620ee1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:47:14,240 [239] [DEBUG] [app] Ending request: urn:request:bd6686fb-677b-4cca-8151-a28b77ff21bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd6686fb-677b-4cca-8151-a28b77ff21bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:47:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:47:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:14,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:47:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:14,243 [237] [DEBUG] [app] Starting request: urn:request:b7c6882c-c8af-4780-8577-982e02540c7c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:14,243 [236] [DEBUG] [app] Starting request: urn:request:01cf6d6a-18a8-4f36-b265-7c66640c802c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:14,244 [237] [DEBUG] [app] Ending request: urn:request:b7c6882c-c8af-4780-8577-982e02540c7c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7c6882c-c8af-4780-8577-982e02540c7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:47:14,244 [236] [DEBUG] [app] Ending request: urn:request:01cf6d6a-18a8-4f36-b265-7c66640c802c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01cf6d6a-18a8-4f36-b265-7c66640c802c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:47:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:47:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:14,255 [235] [DEBUG] [app] Ending request: urn:request:cabd21bf-8c11-483c-9976-5205f17966e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cabd21bf-8c11-483c-9976-5205f17966e6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:14,255 [238] [DEBUG] [app] Ending request: urn:request:c9f3784e-b0fe-4538-a8d0-c7d6d36acda7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9f3784e-b0fe-4538-a8d0-c7d6d36acda7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:47:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:47:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:47:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
notificationworker stdout | 2025-02-07 04:47:15,066 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:47:15,093 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:47:16,244 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:16,677 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:47:17,052 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:47:17,055 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:47:17,059 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:47:17,062 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:47:17,064 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:47:17,234 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:47:17,237 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:47:17,367 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:47:17,830 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:47:18,075 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:47:18,078 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:47:18,081 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:47:18,244 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:47:18,437 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:47:18,534 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:47:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:28.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 04:47:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:28 UTC)" (scheduled at 2025-02-07 04:47:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:47:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:47:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 18, 732859), True, datetime.datetime(2025, 2, 7, 4, 47, 18, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:47:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:47:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:47:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:47:19,328 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:47:19,330 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:47:19,334 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:47:19,338 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:47:19,340 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:47:19,343 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:47:19,345 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:47:19,348 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:47:19,351 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
buildlogsarchiver stdout | 2025-02-07 04:47:20,079 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:47:20,102 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:47:20,550 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:47:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:47:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:03.419110+00:00 (in 39.997759 seconds)
repositorygcworker stdout | 2025-02-07 04:47:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:23 UTC)" (scheduled at 2025-02-07 04:47:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:47:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:47:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:47:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:47:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:40.424250+00:00 (in 16.000754 seconds)
gcworker stdout | 2025-02-07 04:47:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:54 UTC)" (scheduled at 2025-02-07 04:47:24.423055+00:00)
gcworker stdout | 2025-02-07 04:47:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:47:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:47:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:47:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:33.814166+00:00 (in 5.081579 seconds)
notificationworker stdout | 2025-02-07 04:47:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:38 UTC)" (scheduled at 2025-02-07 04:47:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:47:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:47:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 28, 732899), True, datetime.datetime(2025, 2, 7, 4, 47, 28, 732899), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:47:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:47:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:47:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:47:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:47:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:44.225378+00:00 (in 14.999520 seconds)
quotatotalworker stdout | 2025-02-07 04:47:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:44 UTC)" (scheduled at 2025-02-07 04:47:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:47:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:47:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:47:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:47:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:47:29,236 [238] [DEBUG] [app] Starting request: urn:request:1adc0d6d-4a15-4707-a601-f9af3a74a5ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:29,236 [235] [DEBUG] [app] Starting request: urn:request:cb17d331-7692-4052-a24f-c04b1e3a908f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:47:29,240 [244] [DEBUG] [app] Starting request: urn:request:b37e7789-d05f-4380-a0d8-09e45bcbe666 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:29,240 [239] [DEBUG] [app] Starting request: urn:request:734294eb-72d9-4c62-83ad-7a02e8c58c98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:29,240 [244] [DEBUG] [app] Ending request: urn:request:b37e7789-d05f-4380-a0d8-09e45bcbe666 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b37e7789-d05f-4380-a0d8-09e45bcbe666', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:47:29,240 [239] [DEBUG] [app] Ending request: urn:request:734294eb-72d9-4c62-83ad-7a02e8c58c98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:734294eb-72d9-4c62-83ad-7a02e8c58c98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:47:29,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:47:29,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:29,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:29,244 [236] [DEBUG] [app] Starting request: urn:request:725ec339-9bbf-4d77-b2b3-cfc843fb02f4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:29,244 [237] [DEBUG] [app] Starting request: urn:request:68a85a26-eccb-4ba9-9224-15a6fdddfe61 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:29,244 [236] [DEBUG] [app] Ending request: urn:request:725ec339-9bbf-4d77-b2b3-cfc843fb02f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:725ec339-9bbf-4d77-b2b3-cfc843fb02f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:47:29,244 [237] [DEBUG] [app] Ending request: urn:request:68a85a26-eccb-4ba9-9224-15a6fdddfe61 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68a85a26-eccb-4ba9-9224-15a6fdddfe61', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:47:29,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:29,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:29,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:29,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:29,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:29,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:29,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:29,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:29,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:29,257 [238] [DEBUG] [app] Ending request: urn:request:1adc0d6d-4a15-4707-a601-f9af3a74a5ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1adc0d6d-4a15-4707-a601-f9af3a74a5ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:29,257 [235] [DEBUG] [app] Ending request: urn:request:cb17d331-7692-4052-a24f-c04b1e3a908f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb17d331-7692-4052-a24f-c04b1e3a908f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:29,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:47:29,257 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 04:47:29,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:47:29,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 04:47:29,554 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:47:31,274 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:47:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:47:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:32.625945+00:00 (in 1.004409 seconds)
namespacegcworker stdout | 2025-02-07 04:47:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:31 UTC)" (scheduled at 2025-02-07 04:47:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:47:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:47:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 31, 621802), True, datetime.datetime(2025, 2, 7, 4, 47, 31, 621802), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:47:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:47:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:47:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:47:32,441 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:47:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:47:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:31.621131+00:00 (in 58.994740 seconds)
namespacegcworker stdout | 2025-02-07 04:47:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:32 UTC)" (scheduled at 2025-02-07 04:47:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:47:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:47:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:47:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:38.732147+00:00 (in 4.917549 seconds)
notificationworker stdout | 2025-02-07 04:47:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:52:33 UTC)" (scheduled at 2025-02-07 04:47:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:47:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 47, 33, 814869), 'notification/%'])
notificationworker stdout | 2025-02-07 04:47:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 47, 33, 814869), True, datetime.datetime(2025, 2, 7, 4, 47, 33, 814869), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:47:33,824 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 47, 33, 814869), True, datetime.datetime(2025, 2, 7, 4, 47, 33, 814869), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 47, 33, 814869), 'notification/%'])
notificationworker stdout | 2025-02-07 04:47:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:47:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:52:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:47:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:47:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:03.919286+00:00 (in 29.999532 seconds)
buildlogsarchiver stdout | 2025-02-07 04:47:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:03 UTC)" (scheduled at 2025-02-07 04:47:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:47:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 47, 33, 920003), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:47:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:47:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:47:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:47:36,756 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:56.713547+00:00 (in 18.997206 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:37 UTC)" (scheduled at 2025-02-07 04:47:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:47:37,830 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:47:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:48.732147+00:00 (in 9.999614 seconds)
notificationworker stdout | 2025-02-07 04:47:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:48 UTC)" (scheduled at 2025-02-07 04:47:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:47:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:47:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 38, 732843), True, datetime.datetime(2025, 2, 7, 4, 47, 38, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:47:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:47:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:47:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:47:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:47:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:54.423055+00:00 (in 13.998381 seconds)
gcworker stdout | 2025-02-07 04:47:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:10 UTC)" (scheduled at 2025-02-07 04:47:40.424250+00:00)
gcworker stdout | 2025-02-07 04:47:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:47:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903360431, None, 1, 0])
gcworker stdout | 2025-02-07 04:47:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:47:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:10 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:47:41,796 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:47:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:47:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:11.831521+00:00 (in 29.999532 seconds)
autopruneworker stdout | 2025-02-07 04:47:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:11 UTC)" (scheduled at 2025-02-07 04:47:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:47:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900061836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:47:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:47:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:47:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:47:42,341 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:47:42,806 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:47:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:47:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:59.225378+00:00 (in 14.999493 seconds)
quotatotalworker stdout | 2025-02-07 04:47:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:59 UTC)" (scheduled at 2025-02-07 04:47:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:47:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:47:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
expiredappspecifictokenworker stdout | 2025-02-07 04:47:44,227 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
quotatotalworker stdout | 2025-02-07 04:47:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:47:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:47:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:47:44,235 [237] [DEBUG] [app] Starting request: urn:request:a6d8d2af-45d0-4500-afc0-45db579f25c9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:44,235 [236] [DEBUG] [app] Starting request: urn:request:e27e8975-27cb-4ca1-a5a4-e678389253e9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:47:44,238 [239] [DEBUG] [app] Starting request: urn:request:9a795ed1-e3d3-43a4-ac4d-3e0921e06916 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:44,238 [243] [DEBUG] [app] Starting request: urn:request:49db7eeb-a106-4a76-aecc-8a66f6cd94a3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:44,239 [243] [DEBUG] [app] Ending request: urn:request:49db7eeb-a106-4a76-aecc-8a66f6cd94a3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49db7eeb-a106-4a76-aecc-8a66f6cd94a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:47:44,239 [239] [DEBUG] [app] Ending request: urn:request:9a795ed1-e3d3-43a4-ac4d-3e0921e06916 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a795ed1-e3d3-43a4-ac4d-3e0921e06916', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:47:44,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:47:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:47:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:44,242 [236] [DEBUG] [app] Starting request: urn:request:3d363d7a-7192-490b-9325-1178987040b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:44,242 [236] [DEBUG] [app] Ending request: urn:request:3d363d7a-7192-490b-9325-1178987040b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d363d7a-7192-490b-9325-1178987040b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:47:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:44,243 [235] [DEBUG] [app] Starting request: urn:request:8d538198-b71c-4ebf-9996-b1dc62cca569 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:44,243 [235] [DEBUG] [app] Ending request: urn:request:8d538198-b71c-4ebf-9996-b1dc62cca569 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d538198-b71c-4ebf-9996-b1dc62cca569', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:47:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:47:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:44,255 [236] [DEBUG] [app] Ending request: urn:request:e27e8975-27cb-4ca1-a5a4-e678389253e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e27e8975-27cb-4ca1-a5a4-e678389253e9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:44,255 [237] [DEBUG] [app] Ending request: urn:request:a6d8d2af-45d0-4500-afc0-45db579f25c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6d8d2af-45d0-4500-afc0-45db579f25c9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:47:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:47:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:47:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
notificationworker stdout | 2025-02-07 04:47:45,098 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:47:45,128 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:47:46,258 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:46,711 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:47:47,061 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:47:47,065 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:47:47,068 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:47:47,071 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:47:47,073 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
servicekey stdout | 2025-02-07 04:47:47,271 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 04:47:47,274 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 04:47:47,403 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:47:47,859 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:47:48,082 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:47:48,085 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:47:48,088 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:47:48,262 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:47:48,444 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:47:48,550 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:47:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:50.814677+00:00 (in 2.082097 seconds)
notificationworker stdout | 2025-02-07 04:47:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:58 UTC)" (scheduled at 2025-02-07 04:47:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:47:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:47:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 48, 732880), True, datetime.datetime(2025, 2, 7, 4, 47, 48, 732880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:47:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:47:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:47:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:47:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:47:49,339 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:47:49,342 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:47:49,346 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:47:49,349 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:47:49,353 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:47:49,355 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:47:49,358 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:47:49,361 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:47:49,364 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:47:50,109 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:47:50,114 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:47:50,578 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:47:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:47:58.732147+00:00 (in 7.917029 seconds)
notificationworker stdout | 2025-02-07 04:47:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:50 UTC)" (scheduled at 2025-02-07 04:47:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:47:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:47:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:47:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:47:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:10.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 04:47:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:24 UTC)" (scheduled at 2025-02-07 04:47:54.423055+00:00)
gcworker stdout | 2025-02-07 04:47:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:47:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:47:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:47:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:56.118855+00:00 (in 59.999516 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:47:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:56 UTC)" (scheduled at 2025-02-07 04:47:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:47:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:47:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:47:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:37.715878+00:00 (in 41.001884 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:56 UTC)" (scheduled at 2025-02-07 04:47:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:47:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:47:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:47:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:08.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 04:47:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:08 UTC)" (scheduled at 2025-02-07 04:47:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:47:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:47:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 47, 58, 732894), True, datetime.datetime(2025, 2, 7, 4, 47, 58, 732894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:47:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:47:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:47:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:47:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:47:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:14.225378+00:00 (in 14.999579 seconds)
quotatotalworker stdout | 2025-02-07 04:47:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:14 UTC)" (scheduled at 2025-02-07 04:47:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:47:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:47:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:47:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:47:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:47:59,236 [235] [DEBUG] [app] Starting request: urn:request:de458000-ef00-4a54-bac8-82f98a0e8ce5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:59,236 [238] [DEBUG] [app] Starting request: urn:request:4b0cd759-cc8a-4209-b23f-e522df17efc6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:47:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:47:59,239 [239] [DEBUG] [app] Starting request: urn:request:7272289c-9a08-4d74-8e86-53f22a33c99b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:59,239 [245] [DEBUG] [app] Starting request: urn:request:2218a04e-0a3b-4c47-b44c-a9e6dbc537f0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:47:59,239 [245] [DEBUG] [app] Ending request: urn:request:2218a04e-0a3b-4c47-b44c-a9e6dbc537f0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2218a04e-0a3b-4c47-b44c-a9e6dbc537f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:47:59,239 [239] [DEBUG] [app] Ending request: urn:request:7272289c-9a08-4d74-8e86-53f22a33c99b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7272289c-9a08-4d74-8e86-53f22a33c99b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:47:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:47:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:47:59,243 [237] [DEBUG] [app] Starting request: urn:request:f18f8318-8667-4d3d-8f18-0063d7d09f11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:59,243 [236] [DEBUG] [app] Starting request: urn:request:a9820db2-45de-4224-96ff-2c2d25f34fb1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:47:59,243 [237] [DEBUG] [app] Ending request: urn:request:f18f8318-8667-4d3d-8f18-0063d7d09f11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f18f8318-8667-4d3d-8f18-0063d7d09f11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:47:59,243 [236] [DEBUG] [app] Ending request: urn:request:a9820db2-45de-4224-96ff-2c2d25f34fb1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9820db2-45de-4224-96ff-2c2d25f34fb1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:47:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:47:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:47:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:47:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:47:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:47:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:47:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:47:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:47:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:47:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:59,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:47:59,255 [235] [DEBUG] [app] Ending request: urn:request:de458000-ef00-4a54-bac8-82f98a0e8ce5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de458000-ef00-4a54-bac8-82f98a0e8ce5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:47:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:47:59,256 [238] [DEBUG] [app] Ending request: urn:request:4b0cd759-cc8a-4209-b23f-e522df17efc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4b0cd759-cc8a-4209-b23f-e522df17efc6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:47:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:47:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:47:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:47:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
storagereplication stdout | 2025-02-07 04:47:59,591 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:47:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:47:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:04.936688+00:00 (in 4.998308 seconds)
exportactionlogsworker stdout | 2025-02-07 04:47:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:59 UTC)" (scheduled at 2025-02-07 04:47:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:47:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:47:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:48:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:48:01,310 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:48:02,477 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:48:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:48:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:23.420886+00:00 (in 20.001294 seconds)
repositorygcworker stdout | 2025-02-07 04:48:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:03 UTC)" (scheduled at 2025-02-07 04:48:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:48:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:48:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 3, 419895), True, datetime.datetime(2025, 2, 7, 4, 48, 3, 419895), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:48:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:48:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:48:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:48:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:48:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:33.919286+00:00 (in 29.999537 seconds)
buildlogsarchiver stdout | 2025-02-07 04:48:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:33 UTC)" (scheduled at 2025-02-07 04:48:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:48:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 48, 3, 920006), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:48:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:48:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:48:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:48:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:48:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:59.937899+00:00 (in 55.000746 seconds)
exportactionlogsworker stdout | 2025-02-07 04:48:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:04 UTC)" (scheduled at 2025-02-07 04:48:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:48:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:48:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 4, 937525), True, datetime.datetime(2025, 2, 7, 4, 48, 4, 937525), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:48:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:48:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:48:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:48:06,778 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:48:07,846 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:48:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:48:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:18.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:48:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:18 UTC)" (scheduled at 2025-02-07 04:48:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:48:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:48:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 8, 732874), True, datetime.datetime(2025, 2, 7, 4, 48, 8, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:48:08,746 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:48:08,746 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:48:08,746 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:48:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:48:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:24.423055+00:00 (in 13.998323 seconds)
gcworker stdout | 2025-02-07 04:48:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:40 UTC)" (scheduled at 2025-02-07 04:48:10.424250+00:00)
gcworker stdout | 2025-02-07 04:48:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:48:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903390431, None, 1, 0])
gcworker stdout | 2025-02-07 04:48:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:48:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:40 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:48:11,821 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:48:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:48:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:41.831521+00:00 (in 29.999531 seconds)
autopruneworker stdout | 2025-02-07 04:48:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:41 UTC)" (scheduled at 2025-02-07 04:48:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:48:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900091836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:48:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:48:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:48:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:41 UTC)" executed successfully
builder stdout | 2025-02-07 04:48:12,378 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:48:12,842 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:48:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:48:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:29.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 04:48:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:29 UTC)" (scheduled at 2025-02-07 04:48:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:48:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:48:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:48:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:48:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:48:14,235 [238] [DEBUG] [app] Starting request: urn:request:e1a7834b-0f81-45f4-8a2a-d4765dab3f5f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:14,235 [235] [DEBUG] [app] Starting request: urn:request:d509085e-8649-4008-b02b-c48199318d90 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:48:14,238 [246] [DEBUG] [app] Starting request: urn:request:54a53ca4-8272-4e80-a654-b6987ce0837b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:14,238 [239] [DEBUG] [app] Starting request: urn:request:6ec5c786-4e79-4ee3-a163-cb8c1b2a644e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:14,239 [239] [DEBUG] [app] Ending request: urn:request:6ec5c786-4e79-4ee3-a163-cb8c1b2a644e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ec5c786-4e79-4ee3-a163-cb8c1b2a644e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:48:14,239 [246] [DEBUG] [app] Ending request: urn:request:54a53ca4-8272-4e80-a654-b6987ce0837b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54a53ca4-8272-4e80-a654-b6987ce0837b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:48:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:48:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:48:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:14,241 [235] [DEBUG] [app] Starting request: urn:request:c61ce5e3-decb-486d-9732-b43696b71dfa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:14,241 [235] [DEBUG] [app] Ending request: urn:request:c61ce5e3-decb-486d-9732-b43696b71dfa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c61ce5e3-decb-486d-9732-b43696b71dfa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:48:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:48:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:14,242 [237] [DEBUG] [app] Starting request: urn:request:7fc78673-08d9-4adc-b1b7-ea409b7a12bb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:14,242 [237] [DEBUG] [app] Ending request: urn:request:7fc78673-08d9-4adc-b1b7-ea409b7a12bb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7fc78673-08d9-4adc-b1b7-ea409b7a12bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:48:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:48:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:48:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:14,252 [235] [DEBUG] [app] Ending request: urn:request:d509085e-8649-4008-b02b-c48199318d90 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d509085e-8649-4008-b02b-c48199318d90', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:48:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:48:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:48:14,254 [238] [DEBUG] [app] Ending request: urn:request:e1a7834b-0f81-45f4-8a2a-d4765dab3f5f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1a7834b-0f81-45f4-8a2a-d4765dab3f5f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:48:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:48:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:48:14,263 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:48:15,131 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:48:15,165 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:48:16,274 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:16,744 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:48:17,068 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:48:17,073 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:48:17,075 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:48:17,079 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:48:17,082 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:48:17,282 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:48:17,285 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:48:17,432 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:48:17,887 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:48:18,089 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:48:18,092 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:48:18,095 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:48:18,298 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:48:18,480 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:48:18,579 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:48:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:48:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:28.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:48:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:28 UTC)" (scheduled at 2025-02-07 04:48:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:48:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:48:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 18, 732856), True, datetime.datetime(2025, 2, 7, 4, 48, 18, 732856), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:48:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:48:18,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:48:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:48:19,350 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:48:19,354 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:48:19,356 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:48:19,359 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:48:19,362 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:48:19,365 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:48:19,368 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:48:19,373 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:48:19,375 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:48:20,140 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:48:20,149 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:48:20,602 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:48:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:48:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:32.420707+00:00 (in 8.999323 seconds)
repositorygcworker stdout | 2025-02-07 04:48:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:23 UTC)" (scheduled at 2025-02-07 04:48:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:48:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:48:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:48:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:48:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:40.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 04:48:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:54 UTC)" (scheduled at 2025-02-07 04:48:24.423055+00:00)
gcworker stdout | 2025-02-07 04:48:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:48:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:48:54 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 04:48:24,735 [63] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2025-02-07 04:48:24,735 [63] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:24.735445+00:00 (in 3599.999552 seconds)
globalpromstats stdout | 2025-02-07 04:48:24,736 [63] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-02-07 05:48:24 UTC)" (scheduled at 2025-02-07 04:48:24.735445+00:00)
globalpromstats stdout | 2025-02-07 04:48:24,736 [63] [DEBUG] [__main__] Attempting to report stats
globalpromstats stdout | 2025-02-07 04:48:24,736 [63] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 04:48:24,736 [63] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ...
globalpromstats stdout | 2025-02-07 04:48:24,739 [63] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2025-02-07 04:48:24,739 [63] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 04:48:24,739 [63] [DEBUG] [__main__] Reporting global stats
globalpromstats stdout | 2025-02-07 04:48:24,740 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2025-02-07 04:48:24,746 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False])
globalpromstats stdout | 2025-02-07 04:48:24,748 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True])
globalpromstats stdout | 2025-02-07 04:48:24,749 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2025-02-07 04:48:24,750 [63] [DEBUG] [data.database] Disconnecting from database.
globalpromstats stdout | 2025-02-07 04:48:24,750 [63] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 04:48:24,750 [63] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2025-02-07 04:48:24,752 [63] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 04:48:24,752 [63] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-02-07 05:48:24 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:48:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:48:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:38.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:48:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:38 UTC)" (scheduled at 2025-02-07 04:48:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:48:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:48:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 28, 732856), True, datetime.datetime(2025, 2, 7, 4, 48, 28, 732856), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:48:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:48:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:48:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:48:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:48:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:44.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 04:48:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:44 UTC)" (scheduled at 2025-02-07 04:48:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:48:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:48:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:48:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:48:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:48:29,235 [235] [DEBUG] [app] Starting request: urn:request:02dcb43c-2df1-41b2-a7a0-1a0ea3d543b4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:29,235 [238] [DEBUG] [app] Starting request: urn:request:07c3ccff-c2fb-4d4b-b80c-3c44f0b9356a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:48:29,238 [245] [DEBUG] [app] Starting request: urn:request:b2c1b0f4-00e1-43aa-887c-7737546f2c8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:29,238 [246] [DEBUG] [app] Starting request: urn:request:aad1398d-fd74-4681-9fdc-bc5a3c1db736 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:29,238 [245] [DEBUG] [app] Ending request: urn:request:b2c1b0f4-00e1-43aa-887c-7737546f2c8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2c1b0f4-00e1-43aa-887c-7737546f2c8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:48:29,239 [246] [DEBUG] [app] Ending request: urn:request:aad1398d-fd74-4681-9fdc-bc5a3c1db736 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aad1398d-fd74-4681-9fdc-bc5a3c1db736', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:48:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:48:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:48:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:29,241 [237] [DEBUG] [app] Starting request: urn:request:9c66b32f-299f-428b-bd9b-960f750f01fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:29,242 [237] [DEBUG] [app] Ending request: urn:request:9c66b32f-299f-428b-bd9b-960f750f01fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9c66b32f-299f-428b-bd9b-960f750f01fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:48:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:48:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:29,242 [236] [DEBUG] [app] Starting request: urn:request:d3808774-7ac1-47db-95d3-878610004a0f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:29,242 [236] [DEBUG] [app] Ending request: urn:request:d3808774-7ac1-47db-95d3-878610004a0f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3808774-7ac1-47db-95d3-878610004a0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:48:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:48:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:48:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:29,253 [235] [DEBUG] [app] Ending request: urn:request:02dcb43c-2df1-41b2-a7a0-1a0ea3d543b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:02dcb43c-2df1-41b2-a7a0-1a0ea3d543b4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:48:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:48:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:48:29,254 [238] [DEBUG] [app] Ending request: urn:request:07c3ccff-c2fb-4d4b-b80c-3c44f0b9356a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07c3ccff-c2fb-4d4b-b80c-3c44f0b9356a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:48:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:48:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
storagereplication stdout | 2025-02-07 04:48:29,619 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:48:31,346 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:48:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:48:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:32.625945+00:00 (in 1.004413 seconds)
namespacegcworker stdout | 2025-02-07 04:48:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:31 UTC)" (scheduled at 2025-02-07 04:48:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:48:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:48:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 31, 621836), True, datetime.datetime(2025, 2, 7, 4, 48, 31, 621836), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:48:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:48:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:48:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 04:48:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:48:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:03.419110+00:00 (in 30.997979 seconds)
repositorygcworker stdout | 2025-02-07 04:48:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:53:32 UTC)" (scheduled at 2025-02-07 04:48:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:48:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 48, 32, 421405), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:48:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 48, 32, 421405), True, datetime.datetime(2025, 2, 7, 4, 48, 32, 421405), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:48:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 48, 32, 421405), True, datetime.datetime(2025, 2, 7, 4, 48, 32, 421405), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 48, 32, 421405), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:48:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:48:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:53:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:48:32,513 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:48:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:48:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:31.621131+00:00 (in 58.994733 seconds)
namespacegcworker stdout | 2025-02-07 04:48:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:32 UTC)" (scheduled at 2025-02-07 04:48:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:48:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:48:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:48:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:48:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:03.919286+00:00 (in 29.999545 seconds)
buildlogsarchiver stdout | 2025-02-07 04:48:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:03 UTC)" (scheduled at 2025-02-07 04:48:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:48:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 48, 33, 919996), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:48:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:48:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:48:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:48:36,814 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:56.713547+00:00 (in 18.997198 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:37 UTC)" (scheduled at 2025-02-07 04:48:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:48:37,882 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:48:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:48:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:48.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 04:48:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:48 UTC)" (scheduled at 2025-02-07 04:48:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:48:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:48:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 38, 732889), True, datetime.datetime(2025, 2, 7, 4, 48, 38, 732889), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:48:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:48:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:48:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:48:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:48:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:54.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 04:48:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:10 UTC)" (scheduled at 2025-02-07 04:48:40.424250+00:00)
gcworker stdout | 2025-02-07 04:48:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:48:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903420430, None, 1, 0])
gcworker stdout | 2025-02-07 04:48:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:48:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:48:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:48:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:11.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 04:48:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:11 UTC)" (scheduled at 2025-02-07 04:48:41.831521+00:00)
quotatotalworker stdout | 2025-02-07 04:48:41,836 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
autopruneworker stdout | 2025-02-07 04:48:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900121836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:48:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:48:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:48:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:11 UTC)" executed successfully
builder stdout | 2025-02-07 04:48:42,414 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:48:42,866 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:48:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:48:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:59.225378+00:00 (in 14.999528 seconds)
quotatotalworker stdout | 2025-02-07 04:48:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:59 UTC)" (scheduled at 2025-02-07 04:48:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:48:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:48:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:48:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:48:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:48:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:48:44,235 [238] [DEBUG] [app] Starting request: urn:request:d7962678-feee-4303-bc66-8e7d732a2aaa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:44,235 [235] [DEBUG] [app] Starting request: urn:request:828e843e-cbff-437b-9941-7117e1513c4a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:48:44,238 [239] [DEBUG] [app] Starting request: urn:request:75316ce9-da3a-45bf-9e52-45baddea3471 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:44,238 [246] [DEBUG] [app] Starting request: urn:request:da8b8687-58a1-4310-8d7f-bb98af369ec8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:44,238 [239] [DEBUG] [app] Ending request: urn:request:75316ce9-da3a-45bf-9e52-45baddea3471 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:75316ce9-da3a-45bf-9e52-45baddea3471', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:48:44,239 [246] [DEBUG] [app] Ending request: urn:request:da8b8687-58a1-4310-8d7f-bb98af369ec8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da8b8687-58a1-4310-8d7f-bb98af369ec8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:48:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:48:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:48:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:48:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:48:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:44,241 [237] [DEBUG] [app] Starting request: urn:request:e03fc4ca-c5d3-4e57-97a8-82b6662646ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:44,242 [237] [DEBUG] [app] Ending request: urn:request:e03fc4ca-c5d3-4e57-97a8-82b6662646ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e03fc4ca-c5d3-4e57-97a8-82b6662646ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:48:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:48:44,242 [235] [DEBUG] [app] Starting request: urn:request:5539511f-e06a-44ba-a48c-d312c535c3c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:44,243 [235] [DEBUG] [app] Ending request: urn:request:5539511f-e06a-44ba-a48c-d312c535c3c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5539511f-e06a-44ba-a48c-d312c535c3c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:48:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:48:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:44,254 [235] [DEBUG] [app] Ending request: urn:request:828e843e-cbff-437b-9941-7117e1513c4a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:828e843e-cbff-437b-9941-7117e1513c4a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:48:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:48:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:48:44,255 [238] [DEBUG] [app] Ending request: urn:request:d7962678-feee-4303-bc66-8e7d732a2aaa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d7962678-feee-4303-bc66-8e7d732a2aaa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 04:48:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:48:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:48:44,299 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:48:45,167 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:48:45,202 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:48:46,300 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:46,774 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:48:47,076 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:48:47,079 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:48:47,084 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:48:47,087 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:48:47,089 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:48:47,318 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:48:47,321 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:48:47,460 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:48:47,901 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:48:48,099 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:48:48,102 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:48:48,104 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:48:48,325 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:48:48,516 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:48:48,594 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:48:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:48:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:50.814677+00:00 (in 2.082097 seconds)
notificationworker stdout | 2025-02-07 04:48:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:58 UTC)" (scheduled at 2025-02-07 04:48:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:48:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:48:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 48, 732883), True, datetime.datetime(2025, 2, 7, 4, 48, 48, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:48:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:48:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:48:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:48:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:48:49,360 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:48:49,363 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:48:49,367 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:48:49,370 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:48:49,373 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:48:49,375 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:48:49,378 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:48:49,383 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:48:49,385 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:48:50,154 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:48:50,185 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:48:50,638 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:48:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:48:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:48:58.732147+00:00 (in 7.916997 seconds)
notificationworker stdout | 2025-02-07 04:48:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:50 UTC)" (scheduled at 2025-02-07 04:48:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:48:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:48:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:48:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:48:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:10.424250+00:00 (in 16.000753 seconds)
gcworker stdout | 2025-02-07 04:48:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:24 UTC)" (scheduled at 2025-02-07 04:48:54.423055+00:00)
gcworker stdout | 2025-02-07 04:48:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:48:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:48:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:48:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:56.118855+00:00 (in 59.999506 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:48:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:56 UTC)" (scheduled at 2025-02-07 04:48:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:48:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:48:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:48:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:37.715878+00:00 (in 41.001862 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:56 UTC)" (scheduled at 2025-02-07 04:48:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:48:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:48:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:48:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:08.732147+00:00 (in 9.999516 seconds)
notificationworker stdout | 2025-02-07 04:48:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:08 UTC)" (scheduled at 2025-02-07 04:48:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:48:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:48:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 48, 58, 732896), True, datetime.datetime(2025, 2, 7, 4, 48, 58, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:48:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:48:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:48:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:48:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:48:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:14.225378+00:00 (in 14.999525 seconds)
quotatotalworker stdout | 2025-02-07 04:48:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:14 UTC)" (scheduled at 2025-02-07 04:48:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:48:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:48:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:48:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:48:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:48:59,235 [238] [DEBUG] [app] Starting request: urn:request:41c0cf3c-15a7-4817-bdee-d27fd396c954 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:59,235 [237] [DEBUG] [app] Starting request: urn:request:85b9d23c-9185-404e-95ac-c17f0534ff24 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:48:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:48:59,238 [245] [DEBUG] [app] Starting request: urn:request:0f74a9f1-4537-499b-b256-9c3dbc4d7e18 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:59,238 [246] [DEBUG] [app] Starting request: urn:request:b1643e87-2a7a-42f4-ae19-3424225b9690 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:48:59,238 [245] [DEBUG] [app] Ending request: urn:request:0f74a9f1-4537-499b-b256-9c3dbc4d7e18 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f74a9f1-4537-499b-b256-9c3dbc4d7e18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:48:59,238 [246] [DEBUG] [app] Ending request: urn:request:b1643e87-2a7a-42f4-ae19-3424225b9690 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1643e87-2a7a-42f4-ae19-3424225b9690', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:48:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:48:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:48:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:48:59,241 [238] [DEBUG] [app] Starting request: urn:request:1ca8eb68-e948-4aa1-a2ec-da06aed16d5f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:59,241 [235] [DEBUG] [app] Starting request: urn:request:3daba444-91b0-465d-8111-6e9184c350be (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:48:59,241 [238] [DEBUG] [app] Ending request: urn:request:1ca8eb68-e948-4aa1-a2ec-da06aed16d5f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1ca8eb68-e948-4aa1-a2ec-da06aed16d5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:48:59,241 [235] [DEBUG] [app] Ending request: urn:request:3daba444-91b0-465d-8111-6e9184c350be (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3daba444-91b0-465d-8111-6e9184c350be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:48:59,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:48:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:48:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:48:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:48:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:48:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:48:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:48:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:48:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:48:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:48:59,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:59,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:48:59,252 [238] [DEBUG] [app] Ending request: urn:request:41c0cf3c-15a7-4817-bdee-d27fd396c954 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:41c0cf3c-15a7-4817-bdee-d27fd396c954', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:48:59,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:48:59,252 [237] [DEBUG] [app] Ending request: urn:request:85b9d23c-9185-404e-95ac-c17f0534ff24 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:85b9d23c-9185-404e-95ac-c17f0534ff24', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:48:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:48:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:48:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:48:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:48:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 04:48:59,653 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:48:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:48:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:04.936688+00:00 (in 4.998286 seconds)
exportactionlogsworker stdout | 2025-02-07 04:48:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:59 UTC)" (scheduled at 2025-02-07 04:48:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:48:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:48:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:49:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:49:01,354 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:49:02,530 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:49:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:49:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:23.420886+00:00 (in 20.001346 seconds)
repositorygcworker stdout | 2025-02-07 04:49:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:03 UTC)" (scheduled at 2025-02-07 04:49:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:49:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:49:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 3, 419815), True, datetime.datetime(2025, 2, 7, 4, 49, 3, 419815), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:49:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:49:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:49:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:49:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:49:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:33.919286+00:00 (in 29.999551 seconds)
buildlogsarchiver stdout | 2025-02-07 04:49:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:33 UTC)" (scheduled at 2025-02-07 04:49:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:49:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 49, 3, 920011), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:49:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:49:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:49:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:49:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:49:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:52.937648+00:00 (in 48.000539 seconds)
exportactionlogsworker stdout | 2025-02-07 04:49:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:04 UTC)" (scheduled at 2025-02-07 04:49:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:49:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:49:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 4, 937397), True, datetime.datetime(2025, 2, 7, 4, 49, 4, 937397), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:49:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:49:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:49:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:49:06,851 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:49:07,902 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:49:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:49:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:18.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:49:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:18 UTC)" (scheduled at 2025-02-07 04:49:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:49:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:49:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 8, 732867), True, datetime.datetime(2025, 2, 7, 4, 49, 8, 732867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:49:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:49:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:49:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:49:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:49:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:24.423055+00:00 (in 13.998388 seconds)
gcworker stdout | 2025-02-07 04:49:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:40 UTC)" (scheduled at 2025-02-07 04:49:10.424250+00:00)
gcworker stdout | 2025-02-07 04:49:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:49:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903450430, None, 1, 0])
gcworker stdout | 2025-02-07 04:49:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:49:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:49:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:49:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:41.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 04:49:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:41 UTC)" (scheduled at 2025-02-07 04:49:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:49:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900151836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:49:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:49:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:49:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:49:11,867 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:49:12,423 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:49:12,882 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:49:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:49:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:29.225378+00:00 (in 14.999482 seconds)
quotatotalworker stdout | 2025-02-07 04:49:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:29 UTC)" (scheduled at 2025-02-07 04:49:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:49:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:49:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:49:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:49:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:49:14,235 [236] [DEBUG] [app] Starting request: urn:request:47092670-695f-41b1-ab32-bf7f27ac2d48 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:14,235 [235] [DEBUG] [app] Starting request: urn:request:889a17d1-3290-40e5-8967-d85cef77e15f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:49:14,239 [239] [DEBUG] [app] Starting request: urn:request:247082d2-7ea5-42c8-a8a5-583ff03401c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:14,239 [246] [DEBUG] [app] Starting request: urn:request:adbf6801-34aa-4c1a-aac9-0f3322384e70 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:14,239 [239] [DEBUG] [app] Ending request: urn:request:247082d2-7ea5-42c8-a8a5-583ff03401c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:247082d2-7ea5-42c8-a8a5-583ff03401c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:49:14,239 [246] [DEBUG] [app] Ending request: urn:request:adbf6801-34aa-4c1a-aac9-0f3322384e70 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:adbf6801-34aa-4c1a-aac9-0f3322384e70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:49:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:49:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:49:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:14,242 [235] [DEBUG] [app] Starting request: urn:request:93cb25e8-9027-4756-b363-6198a29e08ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:49:14,242 [235] [DEBUG] [app] Ending request: urn:request:93cb25e8-9027-4756-b363-6198a29e08ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93cb25e8-9027-4756-b363-6198a29e08ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:49:14,242 [238] [DEBUG] [app] Starting request: urn:request:bb4f2b01-d715-4901-8c25-9025affcc49d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:49:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:14,242 [238] [DEBUG] [app] Ending request: urn:request:bb4f2b01-d715-4901-8c25-9025affcc49d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb4f2b01-d715-4901-8c25-9025affcc49d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:49:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:49:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:14,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:14,253 [235] [DEBUG] [app] Ending request: urn:request:889a17d1-3290-40e5-8967-d85cef77e15f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:889a17d1-3290-40e5-8967-d85cef77e15f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:14,253 [236] [DEBUG] [app] Ending request: urn:request:47092670-695f-41b1-ab32-bf7f27ac2d48 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47092670-695f-41b1-ab32-bf7f27ac2d48', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:49:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:49:14,315 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:49:15,195 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:49:15,238 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:49:16,337 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:16,810 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:49:17,084 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:49:17,088 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:49:17,092 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:49:17,094 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:49:17,097 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:49:17,327 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:49:17,357 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:49:17,496 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:49:17,934 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:49:18,107 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:49:18,110 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:49:18,113 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:49:18,343 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:49:18,552 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:49:18,631 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:49:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:49:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:28.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 04:49:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:28 UTC)" (scheduled at 2025-02-07 04:49:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:49:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:49:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 18, 732857), True, datetime.datetime(2025, 2, 7, 4, 49, 18, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:49:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:49:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:49:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:49:19,370 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:49:19,375 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:49:19,378 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:49:19,381 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:49:19,385 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:49:19,388 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:49:19,391 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:49:19,393 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:49:19,396 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:49:20,190 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:49:20,210 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:49:20,674 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:49:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:49:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:03.419110+00:00 (in 39.997765 seconds)
repositorygcworker stdout | 2025-02-07 04:49:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:23 UTC)" (scheduled at 2025-02-07 04:49:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:49:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:49:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:49:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:49:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:40.424250+00:00 (in 16.000683 seconds)
gcworker stdout | 2025-02-07 04:49:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:54 UTC)" (scheduled at 2025-02-07 04:49:24.423055+00:00)
gcworker stdout | 2025-02-07 04:49:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:49:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:49:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:49:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:49:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:38.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 04:49:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:38 UTC)" (scheduled at 2025-02-07 04:49:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:49:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:49:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 28, 732915), True, datetime.datetime(2025, 2, 7, 4, 49, 28, 732915), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:49:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:49:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:49:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:49:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:49:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:44.225378+00:00 (in 14.999522 seconds)
quotatotalworker stdout | 2025-02-07 04:49:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:44 UTC)" (scheduled at 2025-02-07 04:49:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:49:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:49:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:49:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:49:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:49:29,235 [235] [DEBUG] [app] Starting request: urn:request:cbbe8afb-2da2-42a1-989f-574f0410856d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:29,235 [237] [DEBUG] [app] Starting request: urn:request:8fb22990-e9ae-40e1-956d-f54651375e77 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:49:29,238 [245] [DEBUG] [app] Starting request: urn:request:89a022f7-d870-4813-bbd9-1dbc68f9cae8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:29,238 [239] [DEBUG] [app] Starting request: urn:request:019f1369-10ed-4ad2-a676-c3e5a6e74ca9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:29,239 [239] [DEBUG] [app] Ending request: urn:request:019f1369-10ed-4ad2-a676-c3e5a6e74ca9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:019f1369-10ed-4ad2-a676-c3e5a6e74ca9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:49:29,239 [245] [DEBUG] [app] Ending request: urn:request:89a022f7-d870-4813-bbd9-1dbc68f9cae8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:89a022f7-d870-4813-bbd9-1dbc68f9cae8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:49:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:49:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:29,242 [238] [DEBUG] [app] Starting request: urn:request:a7731c7e-3dbe-47bf-9c7e-46f9f976174f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:49:29,242 [237] [DEBUG] [app] Starting request: urn:request:599d3c2a-f700-42dc-aa96-a0e375d02bb0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:49:29,242 [238] [DEBUG] [app] Ending request: urn:request:a7731c7e-3dbe-47bf-9c7e-46f9f976174f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7731c7e-3dbe-47bf-9c7e-46f9f976174f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:49:29,242 [237] [DEBUG] [app] Ending request: urn:request:599d3c2a-f700-42dc-aa96-a0e375d02bb0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:599d3c2a-f700-42dc-aa96-a0e375d02bb0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:49:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:29,254 [235] [DEBUG] [app] Ending request: urn:request:cbbe8afb-2da2-42a1-989f-574f0410856d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cbbe8afb-2da2-42a1-989f-574f0410856d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:29,254 [237] [DEBUG] [app] Ending request: urn:request:8fb22990-e9ae-40e1-956d-f54651375e77 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8fb22990-e9ae-40e1-956d-f54651375e77', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:49:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 04:49:29,689 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:49:31,370 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:49:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:49:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:32.625945+00:00 (in 1.004384 seconds)
namespacegcworker stdout | 2025-02-07 04:49:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:31 UTC)" (scheduled at 2025-02-07 04:49:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:49:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:49:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 31, 621834), True, datetime.datetime(2025, 2, 7, 4, 49, 31, 621834), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:49:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:49:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:49:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:31 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:49:32,567 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
namespacegcworker stdout | 2025-02-07 04:49:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:49:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:31.621131+00:00 (in 58.994711 seconds)
namespacegcworker stdout | 2025-02-07 04:49:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:32 UTC)" (scheduled at 2025-02-07 04:49:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:49:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:49:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:49:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:49:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:03.919286+00:00 (in 29.999520 seconds)
buildlogsarchiver stdout | 2025-02-07 04:49:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:03 UTC)" (scheduled at 2025-02-07 04:49:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:49:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 49, 33, 920049), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:49:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:49:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:49:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:49:36,887 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:56.713547+00:00 (in 18.997189 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:37 UTC)" (scheduled at 2025-02-07 04:49:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:49:37,938 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:49:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:49:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:48.732147+00:00 (in 9.999483 seconds)
notificationworker stdout | 2025-02-07 04:49:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:48 UTC)" (scheduled at 2025-02-07 04:49:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:49:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:49:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 38, 732978), True, datetime.datetime(2025, 2, 7, 4, 49, 38, 732978), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:49:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:49:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:49:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:49:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:49:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:54.423055+00:00 (in 13.998349 seconds)
gcworker stdout | 2025-02-07 04:49:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:10 UTC)" (scheduled at 2025-02-07 04:49:40.424250+00:00)
gcworker stdout | 2025-02-07 04:49:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:49:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903480431, None, 1, 0])
gcworker stdout | 2025-02-07 04:49:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:49:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:49:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:49:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:11.831521+00:00 (in 29.999564 seconds)
autopruneworker stdout | 2025-02-07 04:49:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:11 UTC)" (scheduled at 2025-02-07 04:49:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:49:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900181836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:49:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:49:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:49:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:49:41,901 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:49:42,432 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:49:42,918 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:49:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:49:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:59.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:49:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:59 UTC)" (scheduled at 2025-02-07 04:49:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:49:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:49:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:49:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:49:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:49:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:49:44,235 [236] [DEBUG] [app] Starting request: urn:request:44b90cfa-e56d-4010-8fec-2f89958f95a8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:44,235 [235] [DEBUG] [app] Starting request: urn:request:fa96ee0c-a56f-47c7-af53-5458bdf0d7ae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:49:44,238 [239] [DEBUG] [app] Starting request: urn:request:a938d8af-76f8-4adf-be3f-bc8c509545b7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:44,238 [244] [DEBUG] [app] Starting request: urn:request:b2bdb0bb-bd60-404d-9389-3e63d3b88d1c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:44,238 [239] [DEBUG] [app] Ending request: urn:request:a938d8af-76f8-4adf-be3f-bc8c509545b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a938d8af-76f8-4adf-be3f-bc8c509545b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:49:44,238 [244] [DEBUG] [app] Ending request: urn:request:b2bdb0bb-bd60-404d-9389-3e63d3b88d1c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2bdb0bb-bd60-404d-9389-3e63d3b88d1c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:49:44,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:49:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:49:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:44,241 [236] [DEBUG] [app] Starting request: urn:request:2cb28b8c-d8aa-442b-93c8-d50be1047f31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:49:44,241 [238] [DEBUG] [app] Starting request: urn:request:24301d5a-4c3a-42bf-b9ea-c6010b891f82 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:49:44,241 [236] [DEBUG] [app] Ending request: urn:request:2cb28b8c-d8aa-442b-93c8-d50be1047f31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2cb28b8c-d8aa-442b-93c8-d50be1047f31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:49:44,242 [238] [DEBUG] [app] Ending request: urn:request:24301d5a-4c3a-42bf-b9ea-c6010b891f82 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:24301d5a-4c3a-42bf-b9ea-c6010b891f82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:49:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:49:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:44,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:44,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:44,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:44,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:44,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:44,253 [236] [DEBUG] [app] Ending request: urn:request:44b90cfa-e56d-4010-8fec-2f89958f95a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:44b90cfa-e56d-4010-8fec-2f89958f95a8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:44,253 [235] [DEBUG] [app] Ending request: urn:request:fa96ee0c-a56f-47c7-af53-5458bdf0d7ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fa96ee0c-a56f-47c7-af53-5458bdf0d7ae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:44,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:49:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:49:44,341 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:49:45,226 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:49:45,275 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:49:46,373 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:46,843 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:49:47,092 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:49:47,095 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:49:47,098 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:49:47,101 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:49:47,104 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:49:47,349 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:49:47,369 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:49:47,527 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:49:47,968 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:49:48,115 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:49:48,119 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:49:48,121 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:49:48,373 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:49:48,588 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:49:48,640 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:49:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:49:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:50.814677+00:00 (in 2.082039 seconds)
notificationworker stdout | 2025-02-07 04:49:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:58 UTC)" (scheduled at 2025-02-07 04:49:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:49:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:49:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 48, 732940), True, datetime.datetime(2025, 2, 7, 4, 49, 48, 732940), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:49:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:49:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:49:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:49:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:49:49,377 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:49:49,387 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:49:49,389 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:49:49,392 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:49:49,395 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:49:49,398 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:49:49,402 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:49:49,404 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:49:49,407 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:49:50,217 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:49:50,244 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:49:50,689 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:49:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:49:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:58.732147+00:00 (in 7.917000 seconds)
notificationworker stdout | 2025-02-07 04:49:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:50 UTC)" (scheduled at 2025-02-07 04:49:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:49:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:49:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:49:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:49:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:49:59.937899+00:00 (in 6.999796 seconds)
exportactionlogsworker stdout | 2025-02-07 04:49:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:54:52 UTC)" (scheduled at 2025-02-07 04:49:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:49:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 49, 52, 938383), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:49:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 49, 52, 938383), True, datetime.datetime(2025, 2, 7, 4, 49, 52, 938383), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:49:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 49, 52, 938383), True, datetime.datetime(2025, 2, 7, 4, 49, 52, 938383), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 49, 52, 938383), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:49:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:49:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:54:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:49:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:49:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:10.424250+00:00 (in 16.000751 seconds)
gcworker stdout | 2025-02-07 04:49:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:24 UTC)" (scheduled at 2025-02-07 04:49:54.423055+00:00)
gcworker stdout | 2025-02-07 04:49:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:49:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:49:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:49:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:56.118855+00:00 (in 59.999497 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:49:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:56 UTC)" (scheduled at 2025-02-07 04:49:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:49:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:49:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:49:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:37.715878+00:00 (in 41.001885 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:56 UTC)" (scheduled at 2025-02-07 04:49:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:49:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:49:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:49:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:08.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 04:49:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:08 UTC)" (scheduled at 2025-02-07 04:49:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:49:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:49:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 49, 58, 732877), True, datetime.datetime(2025, 2, 7, 4, 49, 58, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:49:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:49:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:49:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:49:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:49:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:14.225378+00:00 (in 14.999531 seconds)
quotatotalworker stdout | 2025-02-07 04:49:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:14 UTC)" (scheduled at 2025-02-07 04:49:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:49:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:49:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:49:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:49:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:49:59,235 [235] [DEBUG] [app] Starting request: urn:request:ef38ff61-6d46-407d-af56-45dabbd07143 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:59,235 [238] [DEBUG] [app] Starting request: urn:request:775232cd-7d57-40d5-baa8-4c9a7d976d70 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:49:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:49:59,238 [239] [DEBUG] [app] Starting request: urn:request:45177bf9-d745-4e0a-aed6-72880967ff79 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:59,238 [244] [DEBUG] [app] Starting request: urn:request:c4d6365d-c101-4ccc-89b4-d9fb17a4b95e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:49:59,238 [239] [DEBUG] [app] Ending request: urn:request:45177bf9-d745-4e0a-aed6-72880967ff79 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:45177bf9-d745-4e0a-aed6-72880967ff79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:49:59,238 [244] [DEBUG] [app] Ending request: urn:request:c4d6365d-c101-4ccc-89b4-d9fb17a4b95e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c4d6365d-c101-4ccc-89b4-d9fb17a4b95e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:49:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:49:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:49:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:49:59,241 [238] [DEBUG] [app] Starting request: urn:request:cfb914a8-ed5d-4199-ac16-d5af39a4392c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:49:59,241 [238] [DEBUG] [app] Ending request: urn:request:cfb914a8-ed5d-4199-ac16-d5af39a4392c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cfb914a8-ed5d-4199-ac16-d5af39a4392c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:49:59,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:49:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:59,243 [236] [DEBUG] [app] Starting request: urn:request:5d2d8f15-fdda-46da-b2ec-2528a8e130a8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:49:59,243 [236] [DEBUG] [app] Ending request: urn:request:5d2d8f15-fdda-46da-b2ec-2528a8e130a8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5d2d8f15-fdda-46da-b2ec-2528a8e130a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:49:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:49:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:49:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:49:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:49:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:49:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:49:59,245 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:49:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:49:59,249 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:49:59,252 [235] [DEBUG] [app] Ending request: urn:request:ef38ff61-6d46-407d-af56-45dabbd07143 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ef38ff61-6d46-407d-af56-45dabbd07143', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:59,252 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:59,252 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 04:49:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:49:59,254 [238] [DEBUG] [app] Ending request: urn:request:775232cd-7d57-40d5-baa8-4c9a7d976d70 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:775232cd-7d57-40d5-baa8-4c9a7d976d70', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:49:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:49:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:49:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:49:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 04:49:59,711 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:49:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:49:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:04.936688+00:00 (in 4.998307 seconds)
exportactionlogsworker stdout | 2025-02-07 04:49:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:59 UTC)" (scheduled at 2025-02-07 04:49:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:49:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:49:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:50:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:50:01,401 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:50:02,597 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:50:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:50:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:23.420886+00:00 (in 20.001301 seconds)
repositorygcworker stdout | 2025-02-07 04:50:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:03 UTC)" (scheduled at 2025-02-07 04:50:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:50:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:50:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 3, 419859), True, datetime.datetime(2025, 2, 7, 4, 50, 3, 419859), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:50:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:50:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:50:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:50:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:50:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:33.919286+00:00 (in 29.999489 seconds)
buildlogsarchiver stdout | 2025-02-07 04:50:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:33 UTC)" (scheduled at 2025-02-07 04:50:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:50:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 50, 3, 920085), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:50:03,928 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:50:03,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:50:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:50:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:50:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:59.937899+00:00 (in 55.000763 seconds)
exportactionlogsworker stdout | 2025-02-07 04:50:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:04 UTC)" (scheduled at 2025-02-07 04:50:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:50:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:50:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 4, 937447), True, datetime.datetime(2025, 2, 7, 4, 50, 4, 937447), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:50:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:50:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:50:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:50:06,918 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:50:07,974 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:50:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:50:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:18.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 04:50:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:18 UTC)" (scheduled at 2025-02-07 04:50:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:50:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:50:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 8, 732843), True, datetime.datetime(2025, 2, 7, 4, 50, 8, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:50:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:50:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:50:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:50:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:50:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:24.423055+00:00 (in 13.998381 seconds)
gcworker stdout | 2025-02-07 04:50:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:40 UTC)" (scheduled at 2025-02-07 04:50:10.424250+00:00)
gcworker stdout | 2025-02-07 04:50:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:50:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903510431, None, 1, 0])
gcworker stdout | 2025-02-07 04:50:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:50:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:50:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:50:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:41.831521+00:00 (in 29.999585 seconds)
autopruneworker stdout | 2025-02-07 04:50:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:41 UTC)" (scheduled at 2025-02-07 04:50:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:50:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900211836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:50:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:50:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:50:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:50:11,907 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:50:12,441 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:50:12,938 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:50:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:50:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:29.225378+00:00 (in 14.999527 seconds)
quotatotalworker stdout | 2025-02-07 04:50:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:29 UTC)" (scheduled at 2025-02-07 04:50:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:50:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:50:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:50:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:50:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:50:14,235 [237] [DEBUG] [app] Starting request: urn:request:7aa68d14-cca2-433c-880e-db21861f8154 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:14,235 [238] [DEBUG] [app] Starting request: urn:request:d64ecbfd-652e-4f12-8741-48985ed9cb37 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:50:14,238 [246] [DEBUG] [app] Starting request: urn:request:fc0902d1-443f-4ab9-83ee-531b4049a41d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:14,238 [239] [DEBUG] [app] Starting request: urn:request:4da9c133-2a42-4595-b58c-c8879e8289e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:14,238 [246] [DEBUG] [app] Ending request: urn:request:fc0902d1-443f-4ab9-83ee-531b4049a41d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc0902d1-443f-4ab9-83ee-531b4049a41d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:50:14,238 [239] [DEBUG] [app] Ending request: urn:request:4da9c133-2a42-4595-b58c-c8879e8289e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4da9c133-2a42-4595-b58c-c8879e8289e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:50:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:50:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:14,241 [237] [DEBUG] [app] Starting request: urn:request:a43d98fc-f452-4b5a-bba0-27fbcf00d49e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:14,241 [235] [DEBUG] [app] Starting request: urn:request:e115ebfc-fd82-428c-8fb6-e5e9fe907a3f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:14,242 [237] [DEBUG] [app] Ending request: urn:request:a43d98fc-f452-4b5a-bba0-27fbcf00d49e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a43d98fc-f452-4b5a-bba0-27fbcf00d49e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:50:14,242 [235] [DEBUG] [app] Ending request: urn:request:e115ebfc-fd82-428c-8fb6-e5e9fe907a3f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e115ebfc-fd82-428c-8fb6-e5e9fe907a3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:50:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:50:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:14,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:14,253 [237] [DEBUG] [app] Ending request: urn:request:7aa68d14-cca2-433c-880e-db21861f8154 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7aa68d14-cca2-433c-880e-db21861f8154', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:14,253 [238] [DEBUG] [app] Ending request: urn:request:d64ecbfd-652e-4f12-8741-48985ed9cb37 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d64ecbfd-652e-4f12-8741-48985ed9cb37', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:50:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
expiredappspecifictokenworker stdout | 2025-02-07 04:50:14,365 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:50:15,264 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:50:15,311 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:50:16,406 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:16,879 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:50:17,102 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:50:17,106 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:50:17,108 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:50:17,111 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:50:17,114 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:50:17,368 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:50:17,402 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:50:17,541 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:50:17,993 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:50:18,123 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:50:18,126 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:50:18,129 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:50:18,401 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:50:18,625 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:50:18,677 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:50:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:50:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 04:50:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:28 UTC)" (scheduled at 2025-02-07 04:50:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:50:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:50:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 18, 732876), True, datetime.datetime(2025, 2, 7, 4, 50, 18, 732876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:50:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:50:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:50:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:28 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:50:19,385 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:50:19,398 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:50:19,401 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:50:19,405 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:50:19,410 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:50:19,413 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:50:19,416 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:50:19,419 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:50:19,422 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:50:20,250 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:50:20,271 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:50:20,720 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:50:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:50:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:03.419110+00:00 (in 39.997757 seconds)
repositorygcworker stdout | 2025-02-07 04:50:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:23 UTC)" (scheduled at 2025-02-07 04:50:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:50:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:50:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:50:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:50:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:40.424250+00:00 (in 16.000749 seconds)
gcworker stdout | 2025-02-07 04:50:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:54 UTC)" (scheduled at 2025-02-07 04:50:24.423055+00:00)
gcworker stdout | 2025-02-07 04:50:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:50:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:50:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:50:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:50:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:38.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 04:50:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:38 UTC)" (scheduled at 2025-02-07 04:50:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:50:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:50:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 28, 732855), True, datetime.datetime(2025, 2, 7, 4, 50, 28, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:50:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:50:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:50:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:50:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:50:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:44.225378+00:00 (in 14.999535 seconds)
quotatotalworker stdout | 2025-02-07 04:50:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:44 UTC)" (scheduled at 2025-02-07 04:50:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:50:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:50:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:50:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:50:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:50:29,235 [237] [DEBUG] [app] Starting request: urn:request:38db89e3-2c21-4bfa-b159-d2569ed7cf9b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:29,235 [236] [DEBUG] [app] Starting request: urn:request:b2d6810e-53ae-4323-a579-391132c8df89 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:50:29,238 [239] [DEBUG] [app] Starting request: urn:request:df07e482-10ae-4431-b9f9-3a80525e2ece (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:29,238 [243] [DEBUG] [app] Starting request: urn:request:c5cdfcc8-97db-40a7-b75e-758d210907ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:29,239 [243] [DEBUG] [app] Ending request: urn:request:c5cdfcc8-97db-40a7-b75e-758d210907ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5cdfcc8-97db-40a7-b75e-758d210907ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:50:29,239 [239] [DEBUG] [app] Ending request: urn:request:df07e482-10ae-4431-b9f9-3a80525e2ece (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df07e482-10ae-4431-b9f9-3a80525e2ece', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:50:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:50:29,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 04:50:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:29,242 [237] [DEBUG] [app] Starting request: urn:request:b94adb4e-5a05-44ef-b800-741f4c71519f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:29,242 [237] [DEBUG] [app] Ending request: urn:request:b94adb4e-5a05-44ef-b800-741f4c71519f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b94adb4e-5a05-44ef-b800-741f4c71519f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:50:29,242 [235] [DEBUG] [app] Starting request: urn:request:6a262c28-88cb-47d4-89ec-c83c5a3790a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:29,242 [235] [DEBUG] [app] Ending request: urn:request:6a262c28-88cb-47d4-89ec-c83c5a3790a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a262c28-88cb-47d4-89ec-c83c5a3790a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:50:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:50:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:29,254 [236] [DEBUG] [app] Ending request: urn:request:b2d6810e-53ae-4323-a579-391132c8df89 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2d6810e-53ae-4323-a579-391132c8df89', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:29,254 [237] [DEBUG] [app] Ending request: urn:request:38db89e3-2c21-4bfa-b159-d2569ed7cf9b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38db89e3-2c21-4bfa-b159-d2569ed7cf9b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:50:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
storagereplication stdout | 2025-02-07 04:50:29,721 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:50:31,437 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:50:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:50:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:32.625945+00:00 (in 1.004330 seconds)
namespacegcworker stdout | 2025-02-07 04:50:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:31 UTC)" (scheduled at 2025-02-07 04:50:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:50:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:50:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 31, 621902), True, datetime.datetime(2025, 2, 7, 4, 50, 31, 621902), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:50:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:50:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:50:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:50:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:50:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:31.621131+00:00 (in 58.994683 seconds)
namespacegcworker stdout | 2025-02-07 04:50:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:32 UTC)" (scheduled at 2025-02-07 04:50:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:50:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:50:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:50:32,633 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:50:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:50:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:03.919286+00:00 (in 29.999539 seconds)
buildlogsarchiver stdout | 2025-02-07 04:50:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:03 UTC)" (scheduled at 2025-02-07 04:50:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:50:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 50, 33, 920024), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:50:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:50:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:50:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:50:36,935 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:56.713547+00:00 (in 18.997175 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:37 UTC)" (scheduled at 2025-02-07 04:50:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:50:38,011 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:50:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:50:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:48.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 04:50:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:48 UTC)" (scheduled at 2025-02-07 04:50:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:50:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:50:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 38, 732851), True, datetime.datetime(2025, 2, 7, 4, 50, 38, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:50:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:50:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:50:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:50:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:50:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:54.423055+00:00 (in 13.998336 seconds)
gcworker stdout | 2025-02-07 04:50:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:10 UTC)" (scheduled at 2025-02-07 04:50:40.424250+00:00)
gcworker stdout | 2025-02-07 04:50:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:50:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903540431, None, 1, 0])
gcworker stdout | 2025-02-07 04:50:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:50:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:50:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:50:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:11.831521+00:00 (in 29.999563 seconds)
autopruneworker stdout | 2025-02-07 04:50:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:11 UTC)" (scheduled at 2025-02-07 04:50:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:50:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900241836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:50:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:50:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:50:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:50:41,935 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:50:42,473 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:50:42,974 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:50:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:50:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:59.225378+00:00 (in 14.999470 seconds)
quotatotalworker stdout | 2025-02-07 04:50:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:59 UTC)" (scheduled at 2025-02-07 04:50:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:50:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:50:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:50:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:50:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:50:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:50:44,236 [237] [DEBUG] [app] Starting request: urn:request:18cb6ea4-3a5b-4425-a5df-e926e1d5a47b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:44,236 [235] [DEBUG] [app] Starting request: urn:request:bd02e012-3e7b-4fc8-8e36-6dc0b4d769eb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:50:44,239 [246] [DEBUG] [app] Starting request: urn:request:4188d4b5-bac6-4419-99aa-6deb4d854523 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:44,239 [245] [DEBUG] [app] Starting request: urn:request:08bc4dd4-a27f-46dc-996a-056974b8b96c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:44,239 [246] [DEBUG] [app] Ending request: urn:request:4188d4b5-bac6-4419-99aa-6deb4d854523 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4188d4b5-bac6-4419-99aa-6deb4d854523', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:50:44,239 [245] [DEBUG] [app] Ending request: urn:request:08bc4dd4-a27f-46dc-996a-056974b8b96c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08bc4dd4-a27f-46dc-996a-056974b8b96c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:50:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:50:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:50:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:44,244 [236] [DEBUG] [app] Starting request: urn:request:ce6b9b8b-edb9-44fa-8dd6-12b95d0d827e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:44,244 [238] [DEBUG] [app] Starting request: urn:request:6f046551-3c78-4236-81a0-7d3f1c863fef (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:44,244 [238] [DEBUG] [app] Ending request: urn:request:6f046551-3c78-4236-81a0-7d3f1c863fef (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f046551-3c78-4236-81a0-7d3f1c863fef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:50:44,244 [236] [DEBUG] [app] Ending request: urn:request:ce6b9b8b-edb9-44fa-8dd6-12b95d0d827e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ce6b9b8b-edb9-44fa-8dd6-12b95d0d827e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 159 0.003)
gunicorn-web stdout | 2025-02-07 04:50:44,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:44,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:44,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:44,245 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 159 0.003)
gunicorn-web stdout | 2025-02-07 04:50:44,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:44,246 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:44,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:44,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:44,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:44,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:44,250 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:44,250 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:44,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:44,255 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:44,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:44,256 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:44,257 [235] [DEBUG] [app] Ending request: urn:request:bd02e012-3e7b-4fc8-8e36-6dc0b4d769eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd02e012-3e7b-4fc8-8e36-6dc0b4d769eb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:44,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:44,258 [237] [DEBUG] [app] Ending request: urn:request:18cb6ea4-3a5b-4425-a5df-e926e1d5a47b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:18cb6ea4-3a5b-4425-a5df-e926e1d5a47b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:44,258 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:44,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 04:50:44,258 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:50:44,401 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:50:45,300 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:50:45,327 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:50:46,441 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:46,907 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:50:47,109 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:50:47,113 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:50:47,115 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:50:47,119 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:50:47,122 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:50:47,404 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:50:47,436 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:50:47,577 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:50:48,029 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:50:48,133 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:50:48,135 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:50:48,138 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:50:48,435 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:50:48,662 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:50:48,713 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 04:50:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:50:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:50.814677+00:00 (in 2.081994 seconds)
notificationworker stdout | 2025-02-07 04:50:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:58 UTC)" (scheduled at 2025-02-07 04:50:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:50:48,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:50:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 48, 733065), True, datetime.datetime(2025, 2, 7, 4, 50, 48, 733065), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:50:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:50:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:50:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:50:58 UTC)" executed successfully
gunicorn-registry stdout | 2025-02-07 04:50:49,396 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:50:49,408 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:50:49,411 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:50:49,415 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:50:49,418 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:50:49,422 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:50:49,424 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:50:49,428 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:50:49,430 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:50:50,286 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:50:50,306 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:50:50,757 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:50:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:50:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:50:58.732147+00:00 (in 7.917002 seconds)
notificationworker stdout | 2025-02-07 04:50:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:50 UTC)" (scheduled at 2025-02-07 04:50:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:50:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:50:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:50:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:50:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:10.424250+00:00 (in 16.000705 seconds)
gcworker stdout | 2025-02-07 04:50:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:24 UTC)" (scheduled at 2025-02-07 04:50:54.423055+00:00)
gcworker stdout | 2025-02-07 04:50:54,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 04:50:54,430 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:50:54,430 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:50:54,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:50:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:50:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:56.118855+00:00 (in 59.999527 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:50:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:56 UTC)" (scheduled at 2025-02-07 04:50:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:50:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:50:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:50:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:37.715878+00:00 (in 41.001887 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:56 UTC)" (scheduled at 2025-02-07 04:50:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:50:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:50:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:50:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:08.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 04:50:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:08 UTC)" (scheduled at 2025-02-07 04:50:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:50:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:50:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 50, 58, 732914), True, datetime.datetime(2025, 2, 7, 4, 50, 58, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:50:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:50:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:50:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:50:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:50:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:14.225378+00:00 (in 14.999523 seconds)
quotatotalworker stdout | 2025-02-07 04:50:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:14 UTC)" (scheduled at 2025-02-07 04:50:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:50:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:50:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:50:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:50:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:50:59,235 [238] [DEBUG] [app] Starting request: urn:request:13f31e6c-54fb-4808-9449-be36e2e522fc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:59,235 [237] [DEBUG] [app] Starting request: urn:request:e457e418-53ab-4fb0-9d07-527612d93595 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:50:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:50:59,238 [244] [DEBUG] [app] Starting request: urn:request:4db4b9db-42f5-421f-a46c-781d8fc007c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:59,238 [239] [DEBUG] [app] Starting request: urn:request:c7a46807-6aa4-4f91-a668-cd6587dce15c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:50:59,239 [239] [DEBUG] [app] Ending request: urn:request:c7a46807-6aa4-4f91-a668-cd6587dce15c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c7a46807-6aa4-4f91-a668-cd6587dce15c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:50:59,239 [244] [DEBUG] [app] Ending request: urn:request:4db4b9db-42f5-421f-a46c-781d8fc007c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4db4b9db-42f5-421f-a46c-781d8fc007c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:50:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:50:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:50:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:50:59,242 [236] [DEBUG] [app] Starting request: urn:request:ef6f7f0c-17c3-45a5-aa90-2cc82ca74613 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:59,242 [236] [DEBUG] [app] Ending request: urn:request:ef6f7f0c-17c3-45a5-aa90-2cc82ca74613 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef6f7f0c-17c3-45a5-aa90-2cc82ca74613', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:50:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:59,242 [237] [DEBUG] [app] Starting request: urn:request:89d50df3-4595-44dc-9301-158fffbf1491 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:50:59,243 [237] [DEBUG] [app] Ending request: urn:request:89d50df3-4595-44dc-9301-158fffbf1491 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89d50df3-4595-44dc-9301-158fffbf1491', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:50:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:50:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:50:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:50:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:50:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:50:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:50:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:50:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:50:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:50:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:50:59,254 [237] [DEBUG] [app] Ending request: urn:request:e457e418-53ab-4fb0-9d07-527612d93595 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e457e418-53ab-4fb0-9d07-527612d93595', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 04:50:59,254 [238] [DEBUG] [app] Ending request: urn:request:13f31e6c-54fb-4808-9449-be36e2e522fc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13f31e6c-54fb-4808-9449-be36e2e522fc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:50:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:50:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:50:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:50:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
storagereplication stdout | 2025-02-07 04:50:59,757 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:50:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:50:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:04.936688+00:00 (in 4.998329 seconds)
exportactionlogsworker stdout | 2025-02-07 04:50:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:59 UTC)" (scheduled at 2025-02-07 04:50:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:50:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:50:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:51:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:51:01,472 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:51:02,645 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:51:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:51:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:23.420886+00:00 (in 20.001294 seconds)
repositorygcworker stdout | 2025-02-07 04:51:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:03 UTC)" (scheduled at 2025-02-07 04:51:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:51:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:51:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 3, 419919), True, datetime.datetime(2025, 2, 7, 4, 51, 3, 419919), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:51:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:51:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:51:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:51:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:51:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:33.919286+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2025-02-07 04:51:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:33 UTC)" (scheduled at 2025-02-07 04:51:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:51:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 51, 3, 920069), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:51:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:51:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:51:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:51:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:51:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:59.937899+00:00 (in 55.000708 seconds)
exportactionlogsworker stdout | 2025-02-07 04:51:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:04 UTC)" (scheduled at 2025-02-07 04:51:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:51:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:51:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 4, 937528), True, datetime.datetime(2025, 2, 7, 4, 51, 4, 937528), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:51:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:51:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:51:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:51:06,945 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:51:08,030 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:51:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:51:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:18.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 04:51:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:18 UTC)" (scheduled at 2025-02-07 04:51:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:51:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:51:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 8, 732886), True, datetime.datetime(2025, 2, 7, 4, 51, 8, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:51:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:51:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:51:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:51:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:51:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:24.423055+00:00 (in 13.998366 seconds)
gcworker stdout | 2025-02-07 04:51:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:40 UTC)" (scheduled at 2025-02-07 04:51:10.424250+00:00)
gcworker stdout | 2025-02-07 04:51:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:51:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903570430, None, 1, 0])
gcworker stdout | 2025-02-07 04:51:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:51:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:51:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:51:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:41.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 04:51:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:41 UTC)" (scheduled at 2025-02-07 04:51:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:51:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900271836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:51:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:51:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:51:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:51:11,956 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:51:12,484 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:51:13,010 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:51:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:51:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:29.225378+00:00 (in 14.999550 seconds)
quotatotalworker stdout | 2025-02-07 04:51:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:29 UTC)" (scheduled at 2025-02-07 04:51:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:51:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:51:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:51:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:51:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:51:14,235 [238] [DEBUG] [app] Starting request: urn:request:d59080cd-0df1-40a5-9729-5a96961f1cfb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:14,235 [237] [DEBUG] [app] Starting request: urn:request:3f4befbf-69d0-478a-af83-0617df5b49ec (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:51:14,239 [239] [DEBUG] [app] Starting request: urn:request:859b6419-6252-43d3-bf29-b6f601854eb5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:14,239 [246] [DEBUG] [app] Starting request: urn:request:caa03de4-f382-411e-a57f-7ce8386c1dfb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:14,239 [239] [DEBUG] [app] Ending request: urn:request:859b6419-6252-43d3-bf29-b6f601854eb5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:859b6419-6252-43d3-bf29-b6f601854eb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:51:14,239 [246] [DEBUG] [app] Ending request: urn:request:caa03de4-f382-411e-a57f-7ce8386c1dfb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:caa03de4-f382-411e-a57f-7ce8386c1dfb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:51:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:51:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:14,242 [238] [DEBUG] [app] Starting request: urn:request:4d41d6a5-0893-452c-8235-17182ed9ba59 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:14,243 [238] [DEBUG] [app] Ending request: urn:request:4d41d6a5-0893-452c-8235-17182ed9ba59 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d41d6a5-0893-452c-8235-17182ed9ba59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:51:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:14,243 [235] [DEBUG] [app] Starting request: urn:request:2862bdf2-89c8-40dd-b6ff-7a041a274cff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:14,243 [235] [DEBUG] [app] Ending request: urn:request:2862bdf2-89c8-40dd-b6ff-7a041a274cff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2862bdf2-89c8-40dd-b6ff-7a041a274cff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:51:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:14,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:51:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:14,254 [238] [DEBUG] [app] Ending request: urn:request:d59080cd-0df1-40a5-9729-5a96961f1cfb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d59080cd-0df1-40a5-9729-5a96961f1cfb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:51:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:51:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:51:14,255 [237] [DEBUG] [app] Ending request: urn:request:3f4befbf-69d0-478a-af83-0617df5b49ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f4befbf-69d0-478a-af83-0617df5b49ec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:51:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:51:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:51:14,437 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:51:15,332 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:51:15,362 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:51:16,472 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:16,937 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:51:17,118 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:51:17,122 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:51:17,125 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:51:17,127 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:51:17,130 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:51:17,420 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:51:17,446 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:51:17,614 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:51:18,048 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:51:18,141 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:51:18,144 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:51:18,148 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:51:18,447 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:51:18,693 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
notificationworker stdout | 2025-02-07 04:51:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:51:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:28.732147+00:00 (in 9.999579 seconds)
notificationworker stdout | 2025-02-07 04:51:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:28 UTC)" (scheduled at 2025-02-07 04:51:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:51:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:51:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 18, 732829), True, datetime.datetime(2025, 2, 7, 4, 51, 18, 732829), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:51:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:51:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:51:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:28 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 04:51:18,743 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:51:19,408 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:51:19,419 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:51:19,422 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:51:19,425 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:51:19,427 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:51:19,430 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:51:19,433 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:51:19,435 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:51:19,439 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:51:20,322 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:51:20,342 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:51:20,774 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:51:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:51:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:03.419110+00:00 (in 39.997676 seconds)
repositorygcworker stdout | 2025-02-07 04:51:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:23 UTC)" (scheduled at 2025-02-07 04:51:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:51:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:51:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:51:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:51:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:40.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 04:51:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:54 UTC)" (scheduled at 2025-02-07 04:51:24.423055+00:00)
gcworker stdout | 2025-02-07 04:51:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:51:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:51:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:51:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:51:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:38.732147+00:00 (in 9.999601 seconds)
notificationworker stdout | 2025-02-07 04:51:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:38 UTC)" (scheduled at 2025-02-07 04:51:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:51:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:51:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 28, 732829), True, datetime.datetime(2025, 2, 7, 4, 51, 28, 732829), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:51:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:51:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:51:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:51:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:51:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:44.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 04:51:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:44 UTC)" (scheduled at 2025-02-07 04:51:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:51:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:51:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:51:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:51:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:51:29,236 [235] [DEBUG] [app] Starting request: urn:request:8180b632-9842-4a15-b87b-28a6e1194376 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:29,236 [237] [DEBUG] [app] Starting request: urn:request:13f33c99-b733-4ccf-93a2-52d0a9580792 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:51:29,239 [239] [DEBUG] [app] Starting request: urn:request:61121b6e-05ad-40e7-821c-88d2ca6fb9aa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:29,239 [245] [DEBUG] [app] Starting request: urn:request:171dfd86-015d-4d20-a1fa-9867480da6a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:29,239 [239] [DEBUG] [app] Ending request: urn:request:61121b6e-05ad-40e7-821c-88d2ca6fb9aa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:61121b6e-05ad-40e7-821c-88d2ca6fb9aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:51:29,239 [245] [DEBUG] [app] Ending request: urn:request:171dfd86-015d-4d20-a1fa-9867480da6a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:171dfd86-015d-4d20-a1fa-9867480da6a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:51:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:51:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:51:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:29,242 [237] [DEBUG] [app] Starting request: urn:request:ee256fe8-ef63-4a38-ad77-c3dd125782a5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:29,242 [238] [DEBUG] [app] Starting request: urn:request:b36f97ed-bbd3-45de-8b70-dc6f34af3c4a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:29,243 [237] [DEBUG] [app] Ending request: urn:request:ee256fe8-ef63-4a38-ad77-c3dd125782a5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ee256fe8-ef63-4a38-ad77-c3dd125782a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:51:29,243 [238] [DEBUG] [app] Ending request: urn:request:b36f97ed-bbd3-45de-8b70-dc6f34af3c4a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b36f97ed-bbd3-45de-8b70-dc6f34af3c4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:51:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:51:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:29,256 [235] [DEBUG] [app] Ending request: urn:request:8180b632-9842-4a15-b87b-28a6e1194376 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8180b632-9842-4a15-b87b-28a6e1194376', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:51:29,256 [237] [DEBUG] [app] Ending request: urn:request:13f33c99-b733-4ccf-93a2-52d0a9580792 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13f33c99-b733-4ccf-93a2-52d0a9580792', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:51:29,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:51:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:51:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:51:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 04:51:29,794 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:51:31,508 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:51:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:51:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:32.625945+00:00 (in 1.004286 seconds)
namespacegcworker stdout | 2025-02-07 04:51:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:31 UTC)" (scheduled at 2025-02-07 04:51:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:51:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:51:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 31, 621981), True, datetime.datetime(2025, 2, 7, 4, 51, 31, 621981), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:51:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:51:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:51:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:51:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:51:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:13.624301+00:00 (in 40.997820 seconds)
namespacegcworker stdout | 2025-02-07 04:51:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:32 UTC)" (scheduled at 2025-02-07 04:51:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:51:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:51:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:51:32,656 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:51:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:51:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:03.919286+00:00 (in 29.999516 seconds)
buildlogsarchiver stdout | 2025-02-07 04:51:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:03 UTC)" (scheduled at 2025-02-07 04:51:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:51:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 51, 33, 920017), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:51:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:51:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:51:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:51:36,967 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:56.713547+00:00 (in 18.997194 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:37 UTC)" (scheduled at 2025-02-07 04:51:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:51:38,066 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:51:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:51:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:48.732147+00:00 (in 9.999539 seconds)
notificationworker stdout | 2025-02-07 04:51:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:48 UTC)" (scheduled at 2025-02-07 04:51:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:51:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:51:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 38, 732889), True, datetime.datetime(2025, 2, 7, 4, 51, 38, 732889), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:51:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:51:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:51:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:51:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:51:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:54.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 04:51:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:10 UTC)" (scheduled at 2025-02-07 04:51:40.424250+00:00)
gcworker stdout | 2025-02-07 04:51:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:51:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903600431, None, 1, 0])
gcworker stdout | 2025-02-07 04:51:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:51:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:51:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:51:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:11.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 04:51:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:11 UTC)" (scheduled at 2025-02-07 04:51:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:51:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900301836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:51:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:51:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:51:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:51:41,978 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:51:42,520 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:51:43,024 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:51:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:51:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:59.225378+00:00 (in 14.999572 seconds)
quotatotalworker stdout | 2025-02-07 04:51:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:59 UTC)" (scheduled at 2025-02-07 04:51:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:51:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:51:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:51:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:51:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:51:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:51:44,235 [235] [DEBUG] [app] Starting request: urn:request:955796b7-8700-4f1f-b48e-f0233ab1052c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:44,235 [237] [DEBUG] [app] Starting request: urn:request:831a86b7-041b-47e2-b900-6d5ce79f05e5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:51:44,238 [246] [DEBUG] [app] Starting request: urn:request:9b535653-05fb-4fe0-bd20-6a830f693d51 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:44,238 [239] [DEBUG] [app] Starting request: urn:request:875bc400-7de0-4fff-af1b-e2505e753276 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:44,238 [246] [DEBUG] [app] Ending request: urn:request:9b535653-05fb-4fe0-bd20-6a830f693d51 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b535653-05fb-4fe0-bd20-6a830f693d51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:51:44,238 [239] [DEBUG] [app] Ending request: urn:request:875bc400-7de0-4fff-af1b-e2505e753276 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:875bc400-7de0-4fff-af1b-e2505e753276', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:51:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:51:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:51:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:44,241 [236] [DEBUG] [app] Starting request: urn:request:fafbf932-d659-4195-b789-183f501ab1bf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:44,241 [238] [DEBUG] [app] Starting request: urn:request:8d1131ac-8e0a-44a2-b874-3502b506c3c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:44,242 [238] [DEBUG] [app] Ending request: urn:request:8d1131ac-8e0a-44a2-b874-3502b506c3c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d1131ac-8e0a-44a2-b874-3502b506c3c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:51:44,242 [236] [DEBUG] [app] Ending request: urn:request:fafbf932-d659-4195-b789-183f501ab1bf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fafbf932-d659-4195-b789-183f501ab1bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:51:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:44,254 [237] [DEBUG] [app] Ending request: urn:request:831a86b7-041b-47e2-b900-6d5ce79f05e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:831a86b7-041b-47e2-b900-6d5ce79f05e5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:51:44,254 [235] [DEBUG] [app] Ending request: urn:request:955796b7-8700-4f1f-b48e-f0233ab1052c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:955796b7-8700-4f1f-b48e-f0233ab1052c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:51:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:51:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:51:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:51:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:51:44,465 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:51:45,342 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:51:45,371 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:51:46,504 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:46,968 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:51:47,126 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:51:47,128 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:51:47,131 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:51:47,134 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:51:47,137 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:51:47,429 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:51:47,480 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:51:47,620 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:51:48,068 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:51:48,151 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:51:48,155 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:51:48,157 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:51:48,473 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:51:48,706 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
notificationworker stdout | 2025-02-07 04:51:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:51:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:50.814677+00:00 (in 2.082064 seconds)
notificationworker stdout | 2025-02-07 04:51:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:58 UTC)" (scheduled at 2025-02-07 04:51:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:51:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:51:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 48, 732918), True, datetime.datetime(2025, 2, 7, 4, 51, 48, 732918), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:51:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:51:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:51:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:51:58 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 04:51:48,780 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:51:49,420 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:51:49,428 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:51:49,431 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:51:49,434 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:51:49,438 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:51:49,442 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:51:49,444 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:51:49,447 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:51:49,450 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
buildlogsarchiver stdout | 2025-02-07 04:51:50,355 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:51:50,358 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:51:50,810 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 04:51:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:51:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:51:58.732147+00:00 (in 7.916998 seconds)
notificationworker stdout | 2025-02-07 04:51:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:50 UTC)" (scheduled at 2025-02-07 04:51:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:51:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:51:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:51:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:51:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:10.424250+00:00 (in 16.000754 seconds)
gcworker stdout | 2025-02-07 04:51:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:24 UTC)" (scheduled at 2025-02-07 04:51:54.423055+00:00)
gcworker stdout | 2025-02-07 04:51:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:51:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:51:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:51:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:56.118855+00:00 (in 59.999509 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:51:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:56 UTC)" (scheduled at 2025-02-07 04:51:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:51:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:51:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:51:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:37.715878+00:00 (in 41.001862 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:56 UTC)" (scheduled at 2025-02-07 04:51:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:51:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:51:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:51:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:08.732147+00:00 (in 9.999514 seconds)
notificationworker stdout | 2025-02-07 04:51:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:08 UTC)" (scheduled at 2025-02-07 04:51:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:51:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:51:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 51, 58, 732897), True, datetime.datetime(2025, 2, 7, 4, 51, 58, 732897), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:51:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:51:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:51:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:51:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:51:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:14.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 04:51:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:14 UTC)" (scheduled at 2025-02-07 04:51:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:51:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:51:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:51:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:51:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:51:59,236 [237] [DEBUG] [app] Starting request: urn:request:43557630-8f8d-4136-90ed-c6d5825b16e0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:59,236 [236] [DEBUG] [app] Starting request: urn:request:c2beffd9-f44b-4bad-b3d2-03ff6f173918 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:51:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:51:59,239 [243] [DEBUG] [app] Starting request: urn:request:f0b07477-cdda-45bd-b554-5c115f33b58a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:59,239 [239] [DEBUG] [app] Starting request: urn:request:c64ed96f-f117-43d0-8dee-c34030697c3d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:51:59,239 [243] [DEBUG] [app] Ending request: urn:request:f0b07477-cdda-45bd-b554-5c115f33b58a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0b07477-cdda-45bd-b554-5c115f33b58a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:51:59,239 [239] [DEBUG] [app] Ending request: urn:request:c64ed96f-f117-43d0-8dee-c34030697c3d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c64ed96f-f117-43d0-8dee-c34030697c3d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:51:59,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:51:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:59,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:51:59,242 [237] [DEBUG] [app] Starting request: urn:request:e9f3ee72-44bf-4501-85a7-e990a739b07a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:59,242 [237] [DEBUG] [app] Ending request: urn:request:e9f3ee72-44bf-4501-85a7-e990a739b07a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e9f3ee72-44bf-4501-85a7-e990a739b07a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:51:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:59,243 [238] [DEBUG] [app] Starting request: urn:request:8d75c443-afb5-45de-9410-ad4fef4b700b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:51:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:59,244 [238] [DEBUG] [app] Ending request: urn:request:8d75c443-afb5-45de-9410-ad4fef4b700b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d75c443-afb5-45de-9410-ad4fef4b700b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:51:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:51:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:51:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:51:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:51:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:51:59,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:51:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:51:59,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:51:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:59,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:51:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:59,254 [237] [DEBUG] [app] Ending request: urn:request:43557630-8f8d-4136-90ed-c6d5825b16e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43557630-8f8d-4136-90ed-c6d5825b16e0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:51:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:51:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:51:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:51:59,256 [236] [DEBUG] [app] Ending request: urn:request:c2beffd9-f44b-4bad-b3d2-03ff6f173918 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2beffd9-f44b-4bad-b3d2-03ff6f173918', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:51:59,256 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:51:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:51:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:51:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 04:51:59,829 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:51:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:51:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:04.936688+00:00 (in 4.998258 seconds)
exportactionlogsworker stdout | 2025-02-07 04:51:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:59 UTC)" (scheduled at 2025-02-07 04:51:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:51:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:51:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:52:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:52:01,544 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:52:02,688 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:52:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:52:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:23.420886+00:00 (in 20.001298 seconds)
repositorygcworker stdout | 2025-02-07 04:52:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:03 UTC)" (scheduled at 2025-02-07 04:52:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:52:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:52:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 3, 419881), True, datetime.datetime(2025, 2, 7, 4, 52, 3, 419881), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:52:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:52:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:52:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:52:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:52:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:33.919286+00:00 (in 29.999546 seconds)
buildlogsarchiver stdout | 2025-02-07 04:52:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:33 UTC)" (scheduled at 2025-02-07 04:52:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:52:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 52, 3, 919987), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:52:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:52:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:52:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:52:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:52:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:59.937899+00:00 (in 55.000780 seconds)
exportactionlogsworker stdout | 2025-02-07 04:52:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:04 UTC)" (scheduled at 2025-02-07 04:52:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:52:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:52:04,937 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 4, 937348), True, datetime.datetime(2025, 2, 7, 4, 52, 4, 937348), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:52:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:52:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:52:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:52:07,003 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:52:08,073 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:52:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:18.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 04:52:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:18 UTC)" (scheduled at 2025-02-07 04:52:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:52:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:52:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 8, 732913), True, datetime.datetime(2025, 2, 7, 4, 52, 8, 732913), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:52:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:52:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:52:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:52:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:52:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:24.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 04:52:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:40 UTC)" (scheduled at 2025-02-07 04:52:10.424250+00:00)
gcworker stdout | 2025-02-07 04:52:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:52:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903630431, None, 1, 0])
gcworker stdout | 2025-02-07 04:52:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:52:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:52:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:52:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 04:52:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:41 UTC)" (scheduled at 2025-02-07 04:52:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:52:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900331836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:52:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:52:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:52:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:52:12,014 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:52:12,556 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:52:13,060 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
namespacegcworker stdout | 2025-02-07 04:52:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:52:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:31.621131+00:00 (in 17.996326 seconds)
namespacegcworker stdout | 2025-02-07 04:52:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:57:13 UTC)" (scheduled at 2025-02-07 04:52:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:52:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 52, 13, 625055), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:52:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 52, 13, 625055), True, datetime.datetime(2025, 2, 7, 4, 52, 13, 625055), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:52:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 52, 13, 625055), True, datetime.datetime(2025, 2, 7, 4, 52, 13, 625055), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 52, 13, 625055), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:52:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:52:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:57:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:52:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:52:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:29.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 04:52:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:29 UTC)" (scheduled at 2025-02-07 04:52:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:52:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:52:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:52:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:52:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:52:14,236 [238] [DEBUG] [app] Starting request: urn:request:f74a0111-6fec-4694-a964-d5e986641e95 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:14,236 [237] [DEBUG] [app] Starting request: urn:request:a39619dc-5c96-4fe7-9d12-64ae1d0e14ba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:52:14,238 [239] [DEBUG] [app] Starting request: urn:request:0170965c-9c84-4158-bb95-21df2011e3f4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:14,238 [244] [DEBUG] [app] Starting request: urn:request:9d1098ce-05b6-42ce-ae32-fba8c4c36c3b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:14,239 [239] [DEBUG] [app] Ending request: urn:request:0170965c-9c84-4158-bb95-21df2011e3f4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0170965c-9c84-4158-bb95-21df2011e3f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:52:14,239 [244] [DEBUG] [app] Ending request: urn:request:9d1098ce-05b6-42ce-ae32-fba8c4c36c3b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d1098ce-05b6-42ce-ae32-fba8c4c36c3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:52:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:52:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:52:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:14,242 [238] [DEBUG] [app] Starting request: urn:request:8f6feef4-3c27-4224-9cff-07b0460e8b88 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:52:14,242 [236] [DEBUG] [app] Starting request: urn:request:d3c7d4e8-c673-404c-9ef6-048e3aaf4f09 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:52:14,242 [238] [DEBUG] [app] Ending request: urn:request:8f6feef4-3c27-4224-9cff-07b0460e8b88 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8f6feef4-3c27-4224-9cff-07b0460e8b88', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:52:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:52:14,242 [236] [DEBUG] [app] Ending request: urn:request:d3c7d4e8-c673-404c-9ef6-048e3aaf4f09 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3c7d4e8-c673-404c-9ef6-048e3aaf4f09', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:52:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:52:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:14,254 [238] [DEBUG] [app] Ending request: urn:request:f74a0111-6fec-4694-a964-d5e986641e95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f74a0111-6fec-4694-a964-d5e986641e95', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:14,254 [237] [DEBUG] [app] Ending request: urn:request:a39619dc-5c96-4fe7-9d12-64ae1d0e14ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a39619dc-5c96-4fe7-9d12-64ae1d0e14ba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:52:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:52:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:52:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:52:14,484 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:52:15,354 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:52:15,398 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:52:16,541 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:17,005 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:52:17,135 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:52:17,138 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:52:17,141 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:52:17,144 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:52:17,147 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:52:17,437 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:52:17,516 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:52:17,648 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:52:18,078 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:52:18,159 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:52:18,162 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:52:18,166 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:52:18,493 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:52:18,725 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
notificationworker stdout | 2025-02-07 04:52:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:28.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 04:52:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:28 UTC)" (scheduled at 2025-02-07 04:52:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:52:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:52:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 18, 732842), True, datetime.datetime(2025, 2, 7, 4, 52, 18, 732842), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:52:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:52:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:52:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:28 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 04:52:18,816 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:52:19,430 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:52:19,436 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:52:19,440 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:52:19,443 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:52:19,445 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:52:19,448 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:52:19,454 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:52:19,461 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:52:19,463 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
buildlogsarchiver stdout | 2025-02-07 04:52:20,391 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:52:20,394 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:52:20,836 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:52:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:52:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:03.419110+00:00 (in 39.997758 seconds)
repositorygcworker stdout | 2025-02-07 04:52:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:23 UTC)" (scheduled at 2025-02-07 04:52:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:52:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:52:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:52:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:52:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:40.424250+00:00 (in 16.000757 seconds)
gcworker stdout | 2025-02-07 04:52:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:54 UTC)" (scheduled at 2025-02-07 04:52:24.423055+00:00)
gcworker stdout | 2025-02-07 04:52:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:52:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:52:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:52:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:33.814166+00:00 (in 5.081564 seconds)
notificationworker stdout | 2025-02-07 04:52:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:38 UTC)" (scheduled at 2025-02-07 04:52:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:52:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:52:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 28, 732886), True, datetime.datetime(2025, 2, 7, 4, 52, 28, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:52:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:52:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:52:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:52:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:52:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:44.225378+00:00 (in 14.999550 seconds)
quotatotalworker stdout | 2025-02-07 04:52:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:44 UTC)" (scheduled at 2025-02-07 04:52:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:52:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:52:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:52:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:52:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:52:29,236 [235] [DEBUG] [app] Starting request: urn:request:0d900f22-b897-4f3d-a0a2-d9c850505d73 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:29,236 [238] [DEBUG] [app] Starting request: urn:request:8a6b24fb-3ec0-4975-8f6f-44d8ca62fa3f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:52:29,239 [239] [DEBUG] [app] Starting request: urn:request:e37d491f-3220-41f9-a2f3-0c27286222da (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:29,240 [246] [DEBUG] [app] Starting request: urn:request:cfe0b7bb-a0df-46e6-84b4-40c23e63e79a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:29,240 [246] [DEBUG] [app] Ending request: urn:request:cfe0b7bb-a0df-46e6-84b4-40c23e63e79a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cfe0b7bb-a0df-46e6-84b4-40c23e63e79a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:52:29,240 [239] [DEBUG] [app] Ending request: urn:request:e37d491f-3220-41f9-a2f3-0c27286222da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e37d491f-3220-41f9-a2f3-0c27286222da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:52:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:52:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:52:29,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:29,244 [236] [DEBUG] [app] Starting request: urn:request:31e9acb8-b86a-4af1-8337-45ca8eabb6ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:52:29,244 [237] [DEBUG] [app] Starting request: urn:request:c3148008-15fb-4667-bbb1-94694877a43d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:52:29,245 [236] [DEBUG] [app] Ending request: urn:request:31e9acb8-b86a-4af1-8337-45ca8eabb6ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31e9acb8-b86a-4af1-8337-45ca8eabb6ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:52:29,245 [237] [DEBUG] [app] Ending request: urn:request:c3148008-15fb-4667-bbb1-94694877a43d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c3148008-15fb-4667-bbb1-94694877a43d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:52:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:52:29,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:52:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:29,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:29,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:29,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:29,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:29,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:29,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:29,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:29,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:29,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:29,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:29,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:29,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:29,257 [235] [DEBUG] [app] Ending request: urn:request:0d900f22-b897-4f3d-a0a2-d9c850505d73 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d900f22-b897-4f3d-a0a2-d9c850505d73', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:29,257 [238] [DEBUG] [app] Ending request: urn:request:8a6b24fb-3ec0-4975-8f6f-44d8ca62fa3f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a6b24fb-3ec0-4975-8f6f-44d8ca62fa3f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:29,258 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:52:29,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:52:29,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:52:29,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
storagereplication stdout | 2025-02-07 04:52:29,865 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:52:31,580 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:52:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:52:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:32.625945+00:00 (in 1.004370 seconds)
namespacegcworker stdout | 2025-02-07 04:52:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:31 UTC)" (scheduled at 2025-02-07 04:52:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:52:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:52:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 31, 621833), True, datetime.datetime(2025, 2, 7, 4, 52, 31, 621833), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:52:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:52:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:52:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:52:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:52:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:31.621131+00:00 (in 58.994762 seconds)
namespacegcworker stdout | 2025-02-07 04:52:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:32 UTC)" (scheduled at 2025-02-07 04:52:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:52:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:52:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:52:32,696 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 04:52:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:38.732147+00:00 (in 4.917540 seconds)
notificationworker stdout | 2025-02-07 04:52:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:57:33 UTC)" (scheduled at 2025-02-07 04:52:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:52:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 52, 33, 814843), 'notification/%'])
notificationworker stdout | 2025-02-07 04:52:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 52, 33, 814843), True, datetime.datetime(2025, 2, 7, 4, 52, 33, 814843), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:52:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 52, 33, 814843), True, datetime.datetime(2025, 2, 7, 4, 52, 33, 814843), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 52, 33, 814843), 'notification/%'])
notificationworker stdout | 2025-02-07 04:52:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:52:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:57:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:52:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:52:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:03.919286+00:00 (in 29.999578 seconds)
buildlogsarchiver stdout | 2025-02-07 04:52:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:03 UTC)" (scheduled at 2025-02-07 04:52:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:52:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 52, 33, 919960), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:52:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:52:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:52:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:52:37,020 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:56.713547+00:00 (in 18.997189 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:37 UTC)" (scheduled at 2025-02-07 04:52:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:52:38,110 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:52:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:48.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 04:52:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:48 UTC)" (scheduled at 2025-02-07 04:52:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:52:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:52:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 38, 732900), True, datetime.datetime(2025, 2, 7, 4, 52, 38, 732900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:52:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:52:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:52:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:52:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:52:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:54.423055+00:00 (in 13.998307 seconds)
gcworker stdout | 2025-02-07 04:52:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:10 UTC)" (scheduled at 2025-02-07 04:52:40.424250+00:00)
gcworker stdout | 2025-02-07 04:52:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:52:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903660431, None, 1, 0])
gcworker stdout | 2025-02-07 04:52:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:52:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:52:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:52:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:11.831521+00:00 (in 29.999564 seconds)
autopruneworker stdout | 2025-02-07 04:52:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:11 UTC)" (scheduled at 2025-02-07 04:52:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:52:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900361836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:52:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:52:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:52:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:52:42,021 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:52:42,592 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:52:43,097 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:52:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:52:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:59.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 04:52:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:59 UTC)" (scheduled at 2025-02-07 04:52:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:52:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:52:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:52:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:52:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:52:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:52:44,235 [238] [DEBUG] [app] Starting request: urn:request:dd83ff8c-51ae-47f7-bef1-b3da06b62eb0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:44,235 [235] [DEBUG] [app] Starting request: urn:request:b649fe7e-65dc-4434-9c63-af6a7a5edfe9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:52:44,238 [245] [DEBUG] [app] Starting request: urn:request:7ce1a6ed-ddb7-4d42-a017-b77484f234ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:44,238 [246] [DEBUG] [app] Starting request: urn:request:6a072f09-532d-44bd-8b14-57f5fd73dca7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:44,238 [245] [DEBUG] [app] Ending request: urn:request:7ce1a6ed-ddb7-4d42-a017-b77484f234ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7ce1a6ed-ddb7-4d42-a017-b77484f234ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:52:44,239 [246] [DEBUG] [app] Ending request: urn:request:6a072f09-532d-44bd-8b14-57f5fd73dca7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6a072f09-532d-44bd-8b14-57f5fd73dca7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:52:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:52:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:52:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:44,241 [235] [DEBUG] [app] Starting request: urn:request:8bd27c64-d7a2-454a-ae61-07535d4281b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:52:44,241 [235] [DEBUG] [app] Ending request: urn:request:8bd27c64-d7a2-454a-ae61-07535d4281b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8bd27c64-d7a2-454a-ae61-07535d4281b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:52:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:52:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:44,242 [237] [DEBUG] [app] Starting request: urn:request:5a33c517-2d1a-4f54-8474-4f3729d90fd9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:52:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:44,242 [237] [DEBUG] [app] Ending request: urn:request:5a33c517-2d1a-4f54-8474-4f3729d90fd9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a33c517-2d1a-4f54-8474-4f3729d90fd9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:52:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:52:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:44,253 [235] [DEBUG] [app] Ending request: urn:request:b649fe7e-65dc-4434-9c63-af6a7a5edfe9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b649fe7e-65dc-4434-9c63-af6a7a5edfe9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:52:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:52:44,254 [238] [DEBUG] [app] Ending request: urn:request:dd83ff8c-51ae-47f7-bef1-b3da06b62eb0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dd83ff8c-51ae-47f7-bef1-b3da06b62eb0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:52:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
expiredappspecifictokenworker stdout | 2025-02-07 04:52:44,520 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:52:45,373 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:52:45,417 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:52:46,577 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:47,025 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:52:47,143 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:52:47,146 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:52:47,149 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:52:47,152 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:52:47,155 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:52:47,467 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:52:47,537 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:52:47,674 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:52:48,085 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:52:48,168 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:52:48,172 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:52:48,175 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:52:48,518 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:52:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:50.814677+00:00 (in 2.082090 seconds)
notificationworker stdout | 2025-02-07 04:52:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:58 UTC)" (scheduled at 2025-02-07 04:52:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:52:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:52:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 48, 732910), True, datetime.datetime(2025, 2, 7, 4, 52, 48, 732910), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:52:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:52:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:52:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:52:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:52:48,758 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:52:48,852 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:52:49,440 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:52:49,443 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:52:49,448 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:52:49,454 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:52:49,457 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:52:49,459 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:52:49,463 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:52:49,470 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:52:49,474 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:52:50,407 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:52:50,427 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:52:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:52:58.732147+00:00 (in 7.917022 seconds)
notificationworker stdout | 2025-02-07 04:52:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:50 UTC)" (scheduled at 2025-02-07 04:52:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:52:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:52:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:50 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:52:50,853 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 04:52:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:52:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:10.424250+00:00 (in 16.000716 seconds)
gcworker stdout | 2025-02-07 04:52:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:24 UTC)" (scheduled at 2025-02-07 04:52:54.423055+00:00)
gcworker stdout | 2025-02-07 04:52:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:52:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:52:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:52:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:56.118855+00:00 (in 59.999513 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:52:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:56 UTC)" (scheduled at 2025-02-07 04:52:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:52:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:52:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:52:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:37.715878+00:00 (in 41.001881 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:56 UTC)" (scheduled at 2025-02-07 04:52:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:52:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:52:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:52:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:08.732147+00:00 (in 9.999582 seconds)
notificationworker stdout | 2025-02-07 04:52:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:08 UTC)" (scheduled at 2025-02-07 04:52:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:52:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:52:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 52, 58, 732885), True, datetime.datetime(2025, 2, 7, 4, 52, 58, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:52:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:52:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:52:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:52:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:52:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:14.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 04:52:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:14 UTC)" (scheduled at 2025-02-07 04:52:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:52:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:52:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:52:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:52:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:52:59,235 [235] [DEBUG] [app] Starting request: urn:request:814cf808-5e91-4ddb-961d-3415b58239f3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:59,235 [238] [DEBUG] [app] Starting request: urn:request:e25e8b3b-90a7-4836-8b37-117676725b6f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:52:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:52:59,238 [239] [DEBUG] [app] Starting request: urn:request:9ddf490e-518f-4d67-834f-9b84959b5c9b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:59,238 [245] [DEBUG] [app] Starting request: urn:request:913decda-5849-43ef-860b-b4a2c54b7bb1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:52:59,239 [245] [DEBUG] [app] Ending request: urn:request:913decda-5849-43ef-860b-b4a2c54b7bb1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:913decda-5849-43ef-860b-b4a2c54b7bb1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:52:59,239 [239] [DEBUG] [app] Ending request: urn:request:9ddf490e-518f-4d67-834f-9b84959b5c9b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9ddf490e-518f-4d67-834f-9b84959b5c9b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:52:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:52:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:52:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:52:59,242 [237] [DEBUG] [app] Starting request: urn:request:2f53caaf-50cf-4c70-820e-a7a05fca5e38 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:52:59,242 [237] [DEBUG] [app] Ending request: urn:request:2f53caaf-50cf-4c70-820e-a7a05fca5e38 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f53caaf-50cf-4c70-820e-a7a05fca5e38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:52:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:52:59,242 [235] [DEBUG] [app] Starting request: urn:request:63dab60f-5205-4e27-a067-a124f0959d1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:52:59,243 [235] [DEBUG] [app] Ending request: urn:request:63dab60f-5205-4e27-a067-a124f0959d1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:63dab60f-5205-4e27-a067-a124f0959d1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:52:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:52:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:52:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:52:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:52:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:52:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:52:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:52:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:52:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:52:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:52:59,254 [235] [DEBUG] [app] Ending request: urn:request:814cf808-5e91-4ddb-961d-3415b58239f3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:814cf808-5e91-4ddb-961d-3415b58239f3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:52:59,254 [238] [DEBUG] [app] Ending request: urn:request:e25e8b3b-90a7-4836-8b37-117676725b6f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e25e8b3b-90a7-4836-8b37-117676725b6f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:52:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:52:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 04:52:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:52:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:52:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
storagereplication stdout | 2025-02-07 04:52:59,878 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:52:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:52:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:04.936688+00:00 (in 4.998344 seconds)
exportactionlogsworker stdout | 2025-02-07 04:52:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:59 UTC)" (scheduled at 2025-02-07 04:52:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:52:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:52:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:53:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:53:01,614 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:53:02,721 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:53:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:53:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:23.420886+00:00 (in 20.001345 seconds)
repositorygcworker stdout | 2025-02-07 04:53:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:03 UTC)" (scheduled at 2025-02-07 04:53:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:53:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:53:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 3, 419806), True, datetime.datetime(2025, 2, 7, 4, 53, 3, 419806), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:53:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:53:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:53:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:53:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:53:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:33.919286+00:00 (in 29.999546 seconds)
buildlogsarchiver stdout | 2025-02-07 04:53:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:33 UTC)" (scheduled at 2025-02-07 04:53:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:53:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 53, 3, 919988), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:53:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:53:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:53:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:53:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:53:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:59.937899+00:00 (in 55.000754 seconds)
exportactionlogsworker stdout | 2025-02-07 04:53:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:04 UTC)" (scheduled at 2025-02-07 04:53:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:53:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:53:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 4, 937378), True, datetime.datetime(2025, 2, 7, 4, 53, 4, 937378), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:53:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:53:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:53:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:53:07,055 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:53:08,129 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:53:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:53:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:18.732147+00:00 (in 9.999617 seconds)
notificationworker stdout | 2025-02-07 04:53:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:18 UTC)" (scheduled at 2025-02-07 04:53:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:53:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:53:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 8, 732840), True, datetime.datetime(2025, 2, 7, 4, 53, 8, 732840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:53:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:53:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:53:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:53:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:53:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:24.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 04:53:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:40 UTC)" (scheduled at 2025-02-07 04:53:10.424250+00:00)
gcworker stdout | 2025-02-07 04:53:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:53:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903690430, None, 1, 0])
gcworker stdout | 2025-02-07 04:53:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:53:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:53:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:53:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:41.831521+00:00 (in 29.999541 seconds)
autopruneworker stdout | 2025-02-07 04:53:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:41 UTC)" (scheduled at 2025-02-07 04:53:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:53:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900391836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:53:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:53:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:53:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:53:12,057 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:53:12,601 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:53:13,133 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:53:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:53:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:29.225378+00:00 (in 14.999538 seconds)
quotatotalworker stdout | 2025-02-07 04:53:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:29 UTC)" (scheduled at 2025-02-07 04:53:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:53:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:53:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:53:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:53:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:53:14,236 [238] [DEBUG] [app] Starting request: urn:request:790ba0ef-a0b6-4f49-8448-38e1f53a3751 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:14,236 [235] [DEBUG] [app] Starting request: urn:request:8bc2a9d9-673e-4033-b4ca-f53ec7978bcb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:53:14,239 [239] [DEBUG] [app] Starting request: urn:request:ebef3994-c348-401a-8ec4-d3658e027b5f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:14,239 [245] [DEBUG] [app] Starting request: urn:request:4e14c92c-be59-4ec3-82ff-f8d70be2bf56 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:14,239 [239] [DEBUG] [app] Ending request: urn:request:ebef3994-c348-401a-8ec4-d3658e027b5f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ebef3994-c348-401a-8ec4-d3658e027b5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:53:14,239 [245] [DEBUG] [app] Ending request: urn:request:4e14c92c-be59-4ec3-82ff-f8d70be2bf56 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e14c92c-be59-4ec3-82ff-f8d70be2bf56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:53:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:53:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:53:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:14,242 [235] [DEBUG] [app] Starting request: urn:request:03367eda-8bb3-48db-a873-2878f35ced85 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:14,242 [235] [DEBUG] [app] Ending request: urn:request:03367eda-8bb3-48db-a873-2878f35ced85 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03367eda-8bb3-48db-a873-2878f35ced85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:53:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:53:14,243 [236] [DEBUG] [app] Starting request: urn:request:4e668b67-9ff3-4674-8641-3a7272ce3cae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:14,243 [236] [DEBUG] [app] Ending request: urn:request:4e668b67-9ff3-4674-8641-3a7272ce3cae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4e668b67-9ff3-4674-8641-3a7272ce3cae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:53:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:53:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:14,254 [235] [DEBUG] [app] Ending request: urn:request:8bc2a9d9-673e-4033-b4ca-f53ec7978bcb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8bc2a9d9-673e-4033-b4ca-f53ec7978bcb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:53:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:53:14,255 [238] [DEBUG] [app] Ending request: urn:request:790ba0ef-a0b6-4f49-8448-38e1f53a3751 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:790ba0ef-a0b6-4f49-8448-38e1f53a3751', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:53:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:53:14,530 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:53:15,409 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:53:15,454 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:53:16,590 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:17,050 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:53:17,153 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:53:17,157 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:53:17,160 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:53:17,163 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:53:17,166 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:53:17,489 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:53:17,562 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:53:17,706 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:53:18,118 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:53:18,176 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:53:18,179 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:53:18,183 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:53:18,540 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:53:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:53:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:28.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 04:53:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:28 UTC)" (scheduled at 2025-02-07 04:53:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:53:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:53:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 18, 732847), True, datetime.datetime(2025, 2, 7, 4, 53, 18, 732847), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:53:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:53:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:53:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:28 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:53:18,795 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:53:18,889 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:53:19,449 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:53:19,454 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:53:19,457 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:53:19,464 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:53:19,467 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:53:19,470 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:53:19,472 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:53:19,479 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:53:19,484 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:53:20,414 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:53:20,447 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:53:20,889 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:53:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:53:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:32.420707+00:00 (in 8.999371 seconds)
repositorygcworker stdout | 2025-02-07 04:53:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:23 UTC)" (scheduled at 2025-02-07 04:53:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:53:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:53:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:53:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:53:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:40.424250+00:00 (in 16.000722 seconds)
gcworker stdout | 2025-02-07 04:53:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:54 UTC)" (scheduled at 2025-02-07 04:53:24.423055+00:00)
gcworker stdout | 2025-02-07 04:53:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:53:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:53:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:53:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:53:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:38.732147+00:00 (in 9.999541 seconds)
notificationworker stdout | 2025-02-07 04:53:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:38 UTC)" (scheduled at 2025-02-07 04:53:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:53:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:53:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 28, 732887), True, datetime.datetime(2025, 2, 7, 4, 53, 28, 732887), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:53:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:53:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:53:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:53:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:53:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:44.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 04:53:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:44 UTC)" (scheduled at 2025-02-07 04:53:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:53:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:53:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:53:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:53:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:53:29,235 [235] [DEBUG] [app] Starting request: urn:request:dd5f3627-6522-4d42-8839-900975a25467 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:29,235 [238] [DEBUG] [app] Starting request: urn:request:2ef21d41-9c6c-4e56-977c-27d90dcee3e2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:53:29,238 [246] [DEBUG] [app] Starting request: urn:request:2d87f8cf-d3f1-40a0-9895-92f25c1d3900 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:29,238 [239] [DEBUG] [app] Starting request: urn:request:272195a6-98d5-4592-b537-4fb6e113b598 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:29,238 [246] [DEBUG] [app] Ending request: urn:request:2d87f8cf-d3f1-40a0-9895-92f25c1d3900 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d87f8cf-d3f1-40a0-9895-92f25c1d3900', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:53:29,238 [239] [DEBUG] [app] Ending request: urn:request:272195a6-98d5-4592-b537-4fb6e113b598 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:272195a6-98d5-4592-b537-4fb6e113b598', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:53:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:53:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:53:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:29,241 [236] [DEBUG] [app] Starting request: urn:request:f1b7ad03-bc79-4eff-a188-19d8631cc87b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:29,241 [237] [DEBUG] [app] Starting request: urn:request:c6ba1b0e-58ed-4570-8f33-998eefc80970 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:29,241 [237] [DEBUG] [app] Ending request: urn:request:c6ba1b0e-58ed-4570-8f33-998eefc80970 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6ba1b0e-58ed-4570-8f33-998eefc80970', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:53:29,242 [236] [DEBUG] [app] Ending request: urn:request:f1b7ad03-bc79-4eff-a188-19d8631cc87b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f1b7ad03-bc79-4eff-a188-19d8631cc87b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:53:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:53:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:53:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:53:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:29,253 [235] [DEBUG] [app] Ending request: urn:request:dd5f3627-6522-4d42-8839-900975a25467 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dd5f3627-6522-4d42-8839-900975a25467', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:29,253 [238] [DEBUG] [app] Ending request: urn:request:2ef21d41-9c6c-4e56-977c-27d90dcee3e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2ef21d41-9c6c-4e56-977c-27d90dcee3e2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:53:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:53:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:53:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
storagereplication stdout | 2025-02-07 04:53:29,915 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 04:53:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:53:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:32.625945+00:00 (in 1.004374 seconds)
namespacegcworker stdout | 2025-02-07 04:53:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:31 UTC)" (scheduled at 2025-02-07 04:53:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:53:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:53:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 31, 621773), True, datetime.datetime(2025, 2, 7, 4, 53, 31, 621773), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:53:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:53:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:53:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:53:31,629 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 04:53:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:53:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:03.419110+00:00 (in 30.997936 seconds)
repositorygcworker stdout | 2025-02-07 04:53:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:58:32 UTC)" (scheduled at 2025-02-07 04:53:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:53:32,422 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 53, 32, 421492), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:53:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 53, 32, 421492), True, datetime.datetime(2025, 2, 7, 4, 53, 32, 421492), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:53:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 53, 32, 421492), True, datetime.datetime(2025, 2, 7, 4, 53, 32, 421492), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 53, 32, 421492), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:53:32,432 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:53:32,432 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:58:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:53:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:53:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:31.621131+00:00 (in 58.994660 seconds)
namespacegcworker stdout | 2025-02-07 04:53:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:32 UTC)" (scheduled at 2025-02-07 04:53:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:53:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:53:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:53:32,758 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:53:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:53:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:03.919286+00:00 (in 29.999521 seconds)
buildlogsarchiver stdout | 2025-02-07 04:53:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:03 UTC)" (scheduled at 2025-02-07 04:53:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:53:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 53, 33, 920024), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:53:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:53:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:53:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:53:37,064 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:56.713547+00:00 (in 18.997206 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:37 UTC)" (scheduled at 2025-02-07 04:53:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:53:38,165 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:53:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:53:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:48.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:53:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:48 UTC)" (scheduled at 2025-02-07 04:53:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:53:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:53:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 38, 732895), True, datetime.datetime(2025, 2, 7, 4, 53, 38, 732895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:53:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:53:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:53:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:53:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:53:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:54.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 04:53:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:10 UTC)" (scheduled at 2025-02-07 04:53:40.424250+00:00)
gcworker stdout | 2025-02-07 04:53:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:53:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903720430, None, 1, 0])
gcworker stdout | 2025-02-07 04:53:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:53:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:53:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:53:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:11.831521+00:00 (in 29.999545 seconds)
autopruneworker stdout | 2025-02-07 04:53:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:11 UTC)" (scheduled at 2025-02-07 04:53:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:53:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900421836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:53:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:53:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:53:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:53:42,080 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:53:42,638 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:53:43,155 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:53:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:53:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:59.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 04:53:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:59 UTC)" (scheduled at 2025-02-07 04:53:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:53:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:53:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:53:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:53:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:53:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:53:44,235 [236] [DEBUG] [app] Starting request: urn:request:767d65a6-c760-421c-bd21-a5903434c701 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:44,235 [235] [DEBUG] [app] Starting request: urn:request:7331a6f8-ba31-4694-bf82-bee6a41450db (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:53:44,239 [239] [DEBUG] [app] Starting request: urn:request:6eaf43b9-3618-4077-8758-2683044572d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:44,239 [245] [DEBUG] [app] Starting request: urn:request:6bc6c97d-90df-4cc6-9785-e171f089f66b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:44,239 [245] [DEBUG] [app] Ending request: urn:request:6bc6c97d-90df-4cc6-9785-e171f089f66b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6bc6c97d-90df-4cc6-9785-e171f089f66b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:53:44,239 [239] [DEBUG] [app] Ending request: urn:request:6eaf43b9-3618-4077-8758-2683044572d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6eaf43b9-3618-4077-8758-2683044572d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:53:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:53:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:53:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:44,242 [235] [DEBUG] [app] Starting request: urn:request:7d5c5554-81ec-4558-83eb-f637f37ef861 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:44,242 [235] [DEBUG] [app] Ending request: urn:request:7d5c5554-81ec-4558-83eb-f637f37ef861 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d5c5554-81ec-4558-83eb-f637f37ef861', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:53:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:53:44,243 [238] [DEBUG] [app] Starting request: urn:request:a57461fc-9dbd-49c3-924c-5c73155072af (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:44,243 [238] [DEBUG] [app] Ending request: urn:request:a57461fc-9dbd-49c3-924c-5c73155072af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a57461fc-9dbd-49c3-924c-5c73155072af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:53:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:53:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:44,253 [235] [DEBUG] [app] Ending request: urn:request:7331a6f8-ba31-4694-bf82-bee6a41450db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7331a6f8-ba31-4694-bf82-bee6a41450db', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:53:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:53:44,254 [236] [DEBUG] [app] Ending request: urn:request:767d65a6-c760-421c-bd21-a5903434c701 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:767d65a6-c760-421c-bd21-a5903434c701', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:53:44,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
expiredappspecifictokenworker stdout | 2025-02-07 04:53:44,544 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:53:45,435 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:53:45,491 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:53:46,627 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:47,087 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:53:47,161 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:53:47,165 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:53:47,168 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:53:47,171 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:53:47,173 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:53:47,518 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:53:47,588 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:53:47,730 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:53:48,133 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:53:48,185 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:53:48,187 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:53:48,190 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:53:48,547 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:53:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:53:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:50.814677+00:00 (in 2.082131 seconds)
notificationworker stdout | 2025-02-07 04:53:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:58 UTC)" (scheduled at 2025-02-07 04:53:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:53:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:53:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 48, 732829), True, datetime.datetime(2025, 2, 7, 4, 53, 48, 732829), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:53:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:53:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:53:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:53:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:53:48,831 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:53:48,925 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:53:49,457 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:53:49,464 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:53:49,468 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:53:49,473 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:53:49,477 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:53:49,480 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:53:49,482 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:53:49,489 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:53:49,492 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:53:50,442 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:53:50,482 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:53:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:53:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:53:58.732147+00:00 (in 7.917015 seconds)
notificationworker stdout | 2025-02-07 04:53:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:50 UTC)" (scheduled at 2025-02-07 04:53:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:53:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:53:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:50 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:53:50,923 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 04:53:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:53:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:10.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 04:53:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:24 UTC)" (scheduled at 2025-02-07 04:53:54.423055+00:00)
gcworker stdout | 2025-02-07 04:53:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:53:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:53:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:53:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:56.118855+00:00 (in 59.999505 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:53:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:56 UTC)" (scheduled at 2025-02-07 04:53:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:53:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:53:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:53:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:37.715878+00:00 (in 41.001848 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:56 UTC)" (scheduled at 2025-02-07 04:53:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:53:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:53:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:53:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:08.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:53:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:08 UTC)" (scheduled at 2025-02-07 04:53:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:53:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:53:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 53, 58, 732863), True, datetime.datetime(2025, 2, 7, 4, 53, 58, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:53:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:53:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:53:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:53:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:53:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:14.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:53:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:14 UTC)" (scheduled at 2025-02-07 04:53:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:53:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:53:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:53:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:53:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:53:59,237 [235] [DEBUG] [app] Starting request: urn:request:7a82189d-9ec0-4cd4-b32b-c019f2abca43 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:59,237 [238] [DEBUG] [app] Starting request: urn:request:6526d05c-5db8-4f9b-b57f-7bb84fb56261 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:53:59,239 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:59,239 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:53:59,241 [246] [DEBUG] [app] Starting request: urn:request:ac4c1da8-e88f-4a18-a816-4e306956ce4e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:59,241 [239] [DEBUG] [app] Starting request: urn:request:3f74c4c6-3117-4001-811c-3a16c4f04090 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:53:59,241 [246] [DEBUG] [app] Ending request: urn:request:ac4c1da8-e88f-4a18-a816-4e306956ce4e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac4c1da8-e88f-4a18-a816-4e306956ce4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:53:59,241 [239] [DEBUG] [app] Ending request: urn:request:3f74c4c6-3117-4001-811c-3a16c4f04090 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f74c4c6-3117-4001-811c-3a16c4f04090', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:53:59,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:53:59,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:53:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:59,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:59,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:53:59,246 [237] [DEBUG] [app] Starting request: urn:request:572f0143-c180-4b2f-99af-a6520d322753 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:59,246 [236] [DEBUG] [app] Starting request: urn:request:32d87ffd-0e0d-4579-bfca-ff0217eb29b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:53:59,246 [237] [DEBUG] [app] Ending request: urn:request:572f0143-c180-4b2f-99af-a6520d322753 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:572f0143-c180-4b2f-99af-a6520d322753', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:53:59,246 [236] [DEBUG] [app] Ending request: urn:request:32d87ffd-0e0d-4579-bfca-ff0217eb29b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32d87ffd-0e0d-4579-bfca-ff0217eb29b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 159 0.003)
gunicorn-web stdout | 2025-02-07 04:53:59,247 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:53:59,247 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:53:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:53:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.003 159 0.003)
gunicorn-web stdout | 2025-02-07 04:53:59,247 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:59,247 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:53:59,247 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:59,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:53:59,248 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:59,248 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:53:59,252 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:59,252 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:53:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:53:59,256 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:59,256 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:53:59,258 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:59,258 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:53:59,259 [235] [DEBUG] [app] Ending request: urn:request:7a82189d-9ec0-4cd4-b32b-c019f2abca43 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a82189d-9ec0-4cd4-b32b-c019f2abca43', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:59,259 [238] [DEBUG] [app] Ending request: urn:request:6526d05c-5db8-4f9b-b57f-7bb84fb56261 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6526d05c-5db8-4f9b-b57f-7bb84fb56261', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:53:59,259 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:53:59,259 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:53:59,260 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:53:59,260 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:53:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.025)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:53:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.025)
storagereplication stdout | 2025-02-07 04:53:59,932 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
exportactionlogsworker stdout | 2025-02-07 04:53:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:53:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:04.936688+00:00 (in 4.998345 seconds)
exportactionlogsworker stdout | 2025-02-07 04:53:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:59 UTC)" (scheduled at 2025-02-07 04:53:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:53:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:53:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:54:59 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:54:01,654 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:54:02,794 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:54:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:54:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:23.420886+00:00 (in 20.001317 seconds)
repositorygcworker stdout | 2025-02-07 04:54:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:03 UTC)" (scheduled at 2025-02-07 04:54:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:54:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:54:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 3, 419883), True, datetime.datetime(2025, 2, 7, 4, 54, 3, 419883), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:54:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:54:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:54:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:54:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:54:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:33.919286+00:00 (in 29.999524 seconds)
buildlogsarchiver stdout | 2025-02-07 04:54:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:33 UTC)" (scheduled at 2025-02-07 04:54:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:54:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 54, 3, 920038), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:54:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:54:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:54:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:54:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:54:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:52.937648+00:00 (in 48.000538 seconds)
exportactionlogsworker stdout | 2025-02-07 04:54:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:04 UTC)" (scheduled at 2025-02-07 04:54:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:54:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:54:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 4, 937396), True, datetime.datetime(2025, 2, 7, 4, 54, 4, 937396), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:54:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:54:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:54:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:54:07,100 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:54:08,201 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:54:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:54:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:18.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 04:54:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:18 UTC)" (scheduled at 2025-02-07 04:54:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:54:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:54:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 8, 732881), True, datetime.datetime(2025, 2, 7, 4, 54, 8, 732881), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:54:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:54:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:54:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:54:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:54:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:24.423055+00:00 (in 13.998383 seconds)
gcworker stdout | 2025-02-07 04:54:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:40 UTC)" (scheduled at 2025-02-07 04:54:10.424250+00:00)
gcworker stdout | 2025-02-07 04:54:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:54:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903750430, None, 1, 0])
gcworker stdout | 2025-02-07 04:54:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:54:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:54:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:54:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:41.831521+00:00 (in 29.999556 seconds)
autopruneworker stdout | 2025-02-07 04:54:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:41 UTC)" (scheduled at 2025-02-07 04:54:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:54:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900451836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:54:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:54:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:54:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:54:12,093 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:54:12,674 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:54:13,191 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:54:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:54:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:29.225378+00:00 (in 14.999513 seconds)
quotatotalworker stdout | 2025-02-07 04:54:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:29 UTC)" (scheduled at 2025-02-07 04:54:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:54:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:54:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:54:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:54:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:54:14,236 [235] [DEBUG] [app] Starting request: urn:request:026b5ea7-873e-4ae3-a38c-7f4a61de1951 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:14,236 [238] [DEBUG] [app] Starting request: urn:request:c64dca4d-cc1b-487d-93d6-de1a0ca2f109 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:54:14,238 [246] [DEBUG] [app] Starting request: urn:request:9ad63df2-871a-49f2-a0f5-94c342f647ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:54:14,238 [239] [DEBUG] [app] Starting request: urn:request:20837d20-7a7b-48cc-be8a-36012a57620a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:54:14,239 [246] [DEBUG] [app] Ending request: urn:request:9ad63df2-871a-49f2-a0f5-94c342f647ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9ad63df2-871a-49f2-a0f5-94c342f647ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:54:14,239 [239] [DEBUG] [app] Ending request: urn:request:20837d20-7a7b-48cc-be8a-36012a57620a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20837d20-7a7b-48cc-be8a-36012a57620a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:54:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:54:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:54:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:14,242 [237] [DEBUG] [app] Starting request: urn:request:98588c0a-34c7-4e88-81fa-fce49b42d09f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:14,242 [236] [DEBUG] [app] Starting request: urn:request:acdcc238-c501-4196-9bf4-d350115695e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:14,243 [236] [DEBUG] [app] Ending request: urn:request:acdcc238-c501-4196-9bf4-d350115695e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:acdcc238-c501-4196-9bf4-d350115695e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:54:14,243 [237] [DEBUG] [app] Ending request: urn:request:98588c0a-34c7-4e88-81fa-fce49b42d09f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:98588c0a-34c7-4e88-81fa-fce49b42d09f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:54:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:54:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:54:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:14,255 [235] [DEBUG] [app] Ending request: urn:request:026b5ea7-873e-4ae3-a38c-7f4a61de1951 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:026b5ea7-873e-4ae3-a38c-7f4a61de1951', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:54:14,255 [238] [DEBUG] [app] Ending request: urn:request:c64dca4d-cc1b-487d-93d6-de1a0ca2f109 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c64dca4d-cc1b-487d-93d6-de1a0ca2f109', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:54:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:54:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:54:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:54:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:54:14,580 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:54:15,444 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:54:15,527 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:54:16,641 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:17,118 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:54:17,168 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:54:17,171 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:54:17,175 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:54:17,179 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:54:17,181 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:54:17,546 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:54:17,618 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:54:17,766 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:54:18,151 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:54:18,192 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:54:18,195 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:54:18,198 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:54:18,584 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:54:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:54:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:28.732147+00:00 (in 9.999572 seconds)
notificationworker stdout | 2025-02-07 04:54:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:28 UTC)" (scheduled at 2025-02-07 04:54:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:54:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:54:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 18, 732860), True, datetime.datetime(2025, 2, 7, 4, 54, 18, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:54:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:54:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:54:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:28 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:54:18,853 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:54:18,945 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:54:19,465 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:54:19,473 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:54:19,477 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:54:19,484 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:54:19,487 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:54:19,490 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:54:19,492 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:54:19,497 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:54:19,501 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:54:20,474 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:54:20,517 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:54:20,949 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:54:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:54:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:03.419110+00:00 (in 39.997737 seconds)
repositorygcworker stdout | 2025-02-07 04:54:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:23 UTC)" (scheduled at 2025-02-07 04:54:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:54:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:54:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:54:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:54:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:40.424250+00:00 (in 16.000754 seconds)
gcworker stdout | 2025-02-07 04:54:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:54 UTC)" (scheduled at 2025-02-07 04:54:24.423055+00:00)
gcworker stdout | 2025-02-07 04:54:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:54:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:54:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:54:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:54:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:38.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 04:54:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:38 UTC)" (scheduled at 2025-02-07 04:54:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:54:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:54:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 28, 732882), True, datetime.datetime(2025, 2, 7, 4, 54, 28, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:54:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:54:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:54:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:54:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:54:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:44.225378+00:00 (in 14.999501 seconds)
quotatotalworker stdout | 2025-02-07 04:54:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:44 UTC)" (scheduled at 2025-02-07 04:54:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:54:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:54:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:54:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:54:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:54:29,235 [237] [DEBUG] [app] Starting request: urn:request:be65554d-e4a6-4717-b9d4-5d01c8650818 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:29,235 [235] [DEBUG] [app] Starting request: urn:request:1003adff-8963-4707-b1ee-57a86a5d1ade (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:54:29,238 [239] [DEBUG] [app] Starting request: urn:request:8fb738e6-11d1-4737-99a8-f9b069bac223 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:54:29,239 [239] [DEBUG] [app] Ending request: urn:request:8fb738e6-11d1-4737-99a8-f9b069bac223 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fb738e6-11d1-4737-99a8-f9b069bac223', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:54:29,239 [244] [DEBUG] [app] Starting request: urn:request:8a43b7ae-41ab-428f-a253-e584612f796e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:54:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:54:29,239 [244] [DEBUG] [app] Ending request: urn:request:8a43b7ae-41ab-428f-a253-e584612f796e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a43b7ae-41ab-428f-a253-e584612f796e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:54:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 04:54:29,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 04:54:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:29,242 [238] [DEBUG] [app] Starting request: urn:request:c393cd21-2b83-456f-af96-29a74228ff26 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:29,242 [238] [DEBUG] [app] Ending request: urn:request:c393cd21-2b83-456f-af96-29a74228ff26 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c393cd21-2b83-456f-af96-29a74228ff26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:54:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:54:29,243 [235] [DEBUG] [app] Starting request: urn:request:321cb479-43cc-4ddd-8cca-b1ddda7d40e0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:29,243 [235] [DEBUG] [app] Ending request: urn:request:321cb479-43cc-4ddd-8cca-b1ddda7d40e0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:321cb479-43cc-4ddd-8cca-b1ddda7d40e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:54:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:54:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:29,254 [235] [DEBUG] [app] Ending request: urn:request:1003adff-8963-4707-b1ee-57a86a5d1ade (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1003adff-8963-4707-b1ee-57a86a5d1ade', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:54:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:54:29,255 [237] [DEBUG] [app] Ending request: urn:request:be65554d-e4a6-4717-b9d4-5d01c8650818 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:be65554d-e4a6-4717-b9d4-5d01c8650818', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:54:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:54:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:54:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 04:54:29,942 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 04:54:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:54:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:32.625945+00:00 (in 1.004302 seconds)
namespacegcworker stdout | 2025-02-07 04:54:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:31 UTC)" (scheduled at 2025-02-07 04:54:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:54:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:54:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 31, 621937), True, datetime.datetime(2025, 2, 7, 4, 54, 31, 621937), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:54:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:54:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:54:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:54:31,690 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:54:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:54:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:31.621131+00:00 (in 58.994711 seconds)
namespacegcworker stdout | 2025-02-07 04:54:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:32 UTC)" (scheduled at 2025-02-07 04:54:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:54:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:54:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:54:32,826 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:54:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:54:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:03.919286+00:00 (in 29.999490 seconds)
buildlogsarchiver stdout | 2025-02-07 04:54:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:03 UTC)" (scheduled at 2025-02-07 04:54:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:54:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 54, 33, 920082), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:54:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:54:33,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:54:33,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:54:37,128 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:56.713547+00:00 (in 18.997207 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:37 UTC)" (scheduled at 2025-02-07 04:54:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:54:38,238 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:54:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:54:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:48.732147+00:00 (in 9.999547 seconds)
notificationworker stdout | 2025-02-07 04:54:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:48 UTC)" (scheduled at 2025-02-07 04:54:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:54:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:54:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 38, 732886), True, datetime.datetime(2025, 2, 7, 4, 54, 38, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:54:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:54:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:54:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:54:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:54:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:54.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 04:54:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:10 UTC)" (scheduled at 2025-02-07 04:54:40.424250+00:00)
gcworker stdout | 2025-02-07 04:54:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:54:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903780431, None, 1, 0])
gcworker stdout | 2025-02-07 04:54:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:54:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:54:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:54:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:11.831521+00:00 (in 29.999499 seconds)
autopruneworker stdout | 2025-02-07 04:54:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:11 UTC)" (scheduled at 2025-02-07 04:54:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:54:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900481836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:54:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:54:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:54:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:54:42,102 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:54:42,710 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:54:43,211 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:54:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:54:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:59.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 04:54:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:59 UTC)" (scheduled at 2025-02-07 04:54:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:54:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:54:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:54:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:54:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:54:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:54:44,236 [235] [DEBUG] [app] Starting request: urn:request:f965719d-8af8-4d7a-9888-c5df37ee99c0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:44,236 [238] [DEBUG] [app] Starting request: urn:request:f15f4c5f-9762-40e4-9bc8-be6a496e9544 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:54:44,239 [246] [DEBUG] [app] Starting request: urn:request:4b2a6b80-37c8-4013-a074-ca702445fa1a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:54:44,239 [245] [DEBUG] [app] Starting request: urn:request:5a9e8dd3-bd36-4be8-b63b-fb92869d71b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:54:44,239 [246] [DEBUG] [app] Ending request: urn:request:4b2a6b80-37c8-4013-a074-ca702445fa1a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4b2a6b80-37c8-4013-a074-ca702445fa1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:54:44,239 [245] [DEBUG] [app] Ending request: urn:request:5a9e8dd3-bd36-4be8-b63b-fb92869d71b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a9e8dd3-bd36-4be8-b63b-fb92869d71b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:54:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:54:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:54:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:44,243 [236] [DEBUG] [app] Starting request: urn:request:b61e8c0f-9824-4635-8e46-8619f05fb4f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:44,243 [237] [DEBUG] [app] Starting request: urn:request:d12c3c71-80c0-47c5-bace-707fc4deb646 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:44,243 [236] [DEBUG] [app] Ending request: urn:request:b61e8c0f-9824-4635-8e46-8619f05fb4f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b61e8c0f-9824-4635-8e46-8619f05fb4f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:54:44,243 [237] [DEBUG] [app] Ending request: urn:request:d12c3c71-80c0-47c5-bace-707fc4deb646 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d12c3c71-80c0-47c5-bace-707fc4deb646', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:54:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:54:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:54:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:44,255 [235] [DEBUG] [app] Ending request: urn:request:f965719d-8af8-4d7a-9888-c5df37ee99c0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f965719d-8af8-4d7a-9888-c5df37ee99c0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:54:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:54:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:54:44,255 [238] [DEBUG] [app] Ending request: urn:request:f15f4c5f-9762-40e4-9bc8-be6a496e9544 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f15f4c5f-9762-40e4-9bc8-be6a496e9544', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:54:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:54:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
expiredappspecifictokenworker stdout | 2025-02-07 04:54:44,602 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:54:45,481 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:54:45,561 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:54:46,650 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:47,149 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:54:47,175 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:54:47,180 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:54:47,183 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:54:47,186 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:54:47,189 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:54:47,553 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:54:47,647 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:54:47,800 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:54:48,167 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:54:48,200 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:54:48,203 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:54:48,206 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:54:48,619 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:54:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:54:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:50.814677+00:00 (in 2.082071 seconds)
notificationworker stdout | 2025-02-07 04:54:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:58 UTC)" (scheduled at 2025-02-07 04:54:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:54:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:54:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 48, 732894), True, datetime.datetime(2025, 2, 7, 4, 54, 48, 732894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:54:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:54:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:54:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:54:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:54:48,890 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:54:48,970 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:54:49,478 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:54:49,481 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:54:49,485 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:54:49,491 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:54:49,498 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:54:49,500 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:54:49,503 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:54:49,507 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:54:49,509 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:54:50,510 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:54:50,534 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:54:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:54:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:58.732147+00:00 (in 7.917018 seconds)
notificationworker stdout | 2025-02-07 04:54:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:50 UTC)" (scheduled at 2025-02-07 04:54:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:54:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:54:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:50 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:54:50,985 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
exportactionlogsworker stdout | 2025-02-07 04:54:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:54:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:54:59.937899+00:00 (in 6.999807 seconds)
exportactionlogsworker stdout | 2025-02-07 04:54:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:59:52 UTC)" (scheduled at 2025-02-07 04:54:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:54:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 54, 52, 938289), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:54:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 54, 52, 938289), True, datetime.datetime(2025, 2, 7, 4, 54, 52, 938289), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:54:52,946 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 54, 52, 938289), True, datetime.datetime(2025, 2, 7, 4, 54, 52, 938289), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 54, 52, 938289), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:54:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:54:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 04:59:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:54:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:54:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:10.424250+00:00 (in 16.000696 seconds)
gcworker stdout | 2025-02-07 04:54:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:24 UTC)" (scheduled at 2025-02-07 04:54:54.423055+00:00)
gcworker stdout | 2025-02-07 04:54:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:54:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:54:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:54:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:56.118855+00:00 (in 59.999522 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:54:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:56 UTC)" (scheduled at 2025-02-07 04:54:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:54:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:54:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:54:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:37.715878+00:00 (in 41.001861 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:56 UTC)" (scheduled at 2025-02-07 04:54:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:54:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:54:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:54:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:08.732147+00:00 (in 9.999575 seconds)
notificationworker stdout | 2025-02-07 04:54:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:08 UTC)" (scheduled at 2025-02-07 04:54:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:54:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:54:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 54, 58, 732843), True, datetime.datetime(2025, 2, 7, 4, 54, 58, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:54:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:54:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:54:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:54:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:54:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:14.225378+00:00 (in 14.999538 seconds)
quotatotalworker stdout | 2025-02-07 04:54:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:14 UTC)" (scheduled at 2025-02-07 04:54:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:54:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:54:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:54:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:54:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:54:59,236 [237] [DEBUG] [app] Starting request: urn:request:1f94c461-ca9a-44a7-9117-e4199d6372bd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:59,236 [235] [DEBUG] [app] Starting request: urn:request:ce93e640-b583-4c17-9b3e-804e585db366 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:54:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:54:59,239 [245] [DEBUG] [app] Starting request: urn:request:d62279cf-e4ef-4234-bdd8-3a10060ea208 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:54:59,239 [246] [DEBUG] [app] Starting request: urn:request:f2503e6f-b62d-4af4-b620-cb8224cfa558 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:54:59,239 [245] [DEBUG] [app] Ending request: urn:request:d62279cf-e4ef-4234-bdd8-3a10060ea208 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d62279cf-e4ef-4234-bdd8-3a10060ea208', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:54:59,239 [246] [DEBUG] [app] Ending request: urn:request:f2503e6f-b62d-4af4-b620-cb8224cfa558 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f2503e6f-b62d-4af4-b620-cb8224cfa558', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:54:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:54:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:54:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:54:59,242 [237] [DEBUG] [app] Starting request: urn:request:0b77c886-3353-46ca-b7d6-6dc2e075e6ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:59,242 [238] [DEBUG] [app] Starting request: urn:request:10a7824d-fd28-437d-a195-7a159ee28bc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:54:59,242 [237] [DEBUG] [app] Ending request: urn:request:0b77c886-3353-46ca-b7d6-6dc2e075e6ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b77c886-3353-46ca-b7d6-6dc2e075e6ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:54:59,242 [238] [DEBUG] [app] Ending request: urn:request:10a7824d-fd28-437d-a195-7a159ee28bc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10a7824d-fd28-437d-a195-7a159ee28bc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:54:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:54:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:54:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:54:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:54:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:54:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:54:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:54:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:54:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:54:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:54:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:54:59,254 [237] [DEBUG] [app] Ending request: urn:request:1f94c461-ca9a-44a7-9117-e4199d6372bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1f94c461-ca9a-44a7-9117-e4199d6372bd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:54:59,254 [235] [DEBUG] [app] Ending request: urn:request:ce93e640-b583-4c17-9b3e-804e585db366 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce93e640-b583-4c17-9b3e-804e585db366', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:54:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:54:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:54:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:54:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:54:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:54:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:54:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:54:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:04.936688+00:00 (in 4.998322 seconds)
exportactionlogsworker stdout | 2025-02-07 04:54:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:59 UTC)" (scheduled at 2025-02-07 04:54:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:54:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:54:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:55:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:54:59,978 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:55:01,726 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:55:02,854 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:55:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:55:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:23.420886+00:00 (in 20.001302 seconds)
repositorygcworker stdout | 2025-02-07 04:55:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:03 UTC)" (scheduled at 2025-02-07 04:55:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:55:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:55:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 3, 419909), True, datetime.datetime(2025, 2, 7, 4, 55, 3, 419909), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:55:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:55:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:55:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:55:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:55:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:33.919286+00:00 (in 29.999480 seconds)
buildlogsarchiver stdout | 2025-02-07 04:55:03,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:33 UTC)" (scheduled at 2025-02-07 04:55:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:55:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 55, 3, 920108), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:55:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:55:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:55:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:55:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:55:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:59.937899+00:00 (in 55.000774 seconds)
exportactionlogsworker stdout | 2025-02-07 04:55:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:04 UTC)" (scheduled at 2025-02-07 04:55:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:55:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:55:04,937 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 4, 937359), True, datetime.datetime(2025, 2, 7, 4, 55, 4, 937359), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:55:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:55:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:55:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:55:07,144 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:55:08,274 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:55:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:55:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:18.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 04:55:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:18 UTC)" (scheduled at 2025-02-07 04:55:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:55:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:55:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 8, 732848), True, datetime.datetime(2025, 2, 7, 4, 55, 8, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:55:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:55:08,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:55:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:55:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:55:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:24.423055+00:00 (in 13.998344 seconds)
gcworker stdout | 2025-02-07 04:55:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:40 UTC)" (scheduled at 2025-02-07 04:55:10.424250+00:00)
gcworker stdout | 2025-02-07 04:55:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:55:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903810431, None, 1, 0])
gcworker stdout | 2025-02-07 04:55:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:55:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:55:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:55:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:41.831521+00:00 (in 29.999512 seconds)
autopruneworker stdout | 2025-02-07 04:55:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:41 UTC)" (scheduled at 2025-02-07 04:55:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:55:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900511836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:55:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:55:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:55:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:55:12,134 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:55:12,734 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:55:13,240 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:55:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:55:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:29.225378+00:00 (in 14.999525 seconds)
quotatotalworker stdout | 2025-02-07 04:55:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:29 UTC)" (scheduled at 2025-02-07 04:55:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:55:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:55:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:55:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:55:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:55:14,235 [235] [DEBUG] [app] Starting request: urn:request:d14828a2-0ce1-44d1-88f6-d387ad87b922 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:14,235 [238] [DEBUG] [app] Starting request: urn:request:23941ceb-b918-44de-9243-55270950bdae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:55:14,238 [239] [DEBUG] [app] Starting request: urn:request:6651808b-578c-421b-808a-06dc5f554e9c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:14,238 [244] [DEBUG] [app] Starting request: urn:request:5e3d6415-2751-4d37-88b8-2a2e5b92c3b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:14,239 [239] [DEBUG] [app] Ending request: urn:request:6651808b-578c-421b-808a-06dc5f554e9c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6651808b-578c-421b-808a-06dc5f554e9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:55:14,239 [244] [DEBUG] [app] Ending request: urn:request:5e3d6415-2751-4d37-88b8-2a2e5b92c3b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5e3d6415-2751-4d37-88b8-2a2e5b92c3b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:55:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:55:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:14,241 [235] [DEBUG] [app] Starting request: urn:request:fdc4a0b3-e28c-4f10-a883-b8a8fad05bdf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:14,241 [236] [DEBUG] [app] Starting request: urn:request:af2ce1ea-ba32-4b54-bf00-48e0e9e428a3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:14,242 [235] [DEBUG] [app] Ending request: urn:request:fdc4a0b3-e28c-4f10-a883-b8a8fad05bdf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdc4a0b3-e28c-4f10-a883-b8a8fad05bdf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:55:14,242 [236] [DEBUG] [app] Ending request: urn:request:af2ce1ea-ba32-4b54-bf00-48e0e9e428a3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af2ce1ea-ba32-4b54-bf00-48e0e9e428a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:55:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:55:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:55:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:14,253 [238] [DEBUG] [app] Ending request: urn:request:23941ceb-b918-44de-9243-55270950bdae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23941ceb-b918-44de-9243-55270950bdae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:55:14,253 [235] [DEBUG] [app] Ending request: urn:request:d14828a2-0ce1-44d1-88f6-d387ad87b922 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d14828a2-0ce1-44d1-88f6-d387ad87b922', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:55:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:55:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
expiredappspecifictokenworker stdout | 2025-02-07 04:55:14,638 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:55:15,517 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:55:15,579 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:55:16,678 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:17,159 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:55:17,185 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:55:17,188 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:55:17,191 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:55:17,195 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:55:17,197 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:55:17,589 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:55:17,683 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:55:17,818 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 04:55:18,190 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-secscan stdout | 2025-02-07 04:55:18,207 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:55:18,210 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:55:18,213 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gcworker stdout | 2025-02-07 04:55:18,655 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:55:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:55:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:28.732147+00:00 (in 9.999586 seconds)
notificationworker stdout | 2025-02-07 04:55:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:28 UTC)" (scheduled at 2025-02-07 04:55:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:55:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:55:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 18, 732847), True, datetime.datetime(2025, 2, 7, 4, 55, 18, 732847), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:55:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:55:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:55:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:28 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:55:18,898 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:55:19,006 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:55:19,489 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:55:19,493 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:55:19,496 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:55:19,499 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:55:19,505 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:55:19,509 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:55:19,511 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:55:19,514 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:55:19,516 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
namespacegcworker stdout | 2025-02-07 04:55:20,547 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 04:55:20,568 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 04:55:21,021 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:55:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:55:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:03.419110+00:00 (in 39.997742 seconds)
repositorygcworker stdout | 2025-02-07 04:55:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:23 UTC)" (scheduled at 2025-02-07 04:55:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:55:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:55:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:55:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:55:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:40.424250+00:00 (in 16.000737 seconds)
gcworker stdout | 2025-02-07 04:55:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:54 UTC)" (scheduled at 2025-02-07 04:55:24.423055+00:00)
gcworker stdout | 2025-02-07 04:55:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:55:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:55:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:55:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:55:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:38.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 04:55:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:38 UTC)" (scheduled at 2025-02-07 04:55:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:55:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:55:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 28, 732876), True, datetime.datetime(2025, 2, 7, 4, 55, 28, 732876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:55:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:55:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:55:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:55:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:55:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:44.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 04:55:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:44 UTC)" (scheduled at 2025-02-07 04:55:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:55:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:55:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:55:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:55:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:55:29,236 [235] [DEBUG] [app] Starting request: urn:request:435032b2-4b6b-4fb8-90a0-452004563986 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:29,236 [238] [DEBUG] [app] Starting request: urn:request:3c3d2565-5ea7-4342-a4bc-a69e354fb35e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:55:29,239 [246] [DEBUG] [app] Starting request: urn:request:77e9873b-2dbd-47a2-9cd9-6e7c284bc9ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:29,239 [243] [DEBUG] [app] Starting request: urn:request:6548715f-a0b4-4ab5-b23f-927581839c73 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:29,240 [243] [DEBUG] [app] Ending request: urn:request:6548715f-a0b4-4ab5-b23f-927581839c73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6548715f-a0b4-4ab5-b23f-927581839c73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:55:29,240 [246] [DEBUG] [app] Ending request: urn:request:77e9873b-2dbd-47a2-9cd9-6e7c284bc9ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77e9873b-2dbd-47a2-9cd9-6e7c284bc9ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:55:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:55:29,240 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:29,244 [236] [DEBUG] [app] Starting request: urn:request:4253d892-f86f-4d4c-8e3d-e13d35f50f47 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:29,244 [237] [DEBUG] [app] Starting request: urn:request:0ccb5106-c533-4456-95dc-d7440e765324 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:29,244 [236] [DEBUG] [app] Ending request: urn:request:4253d892-f86f-4d4c-8e3d-e13d35f50f47 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4253d892-f86f-4d4c-8e3d-e13d35f50f47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:55:29,244 [237] [DEBUG] [app] Ending request: urn:request:0ccb5106-c533-4456-95dc-d7440e765324 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0ccb5106-c533-4456-95dc-d7440e765324', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:55:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:29,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:55:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:29,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:29,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:29,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:29,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:29,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:29,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:29,256 [238] [DEBUG] [app] Ending request: urn:request:3c3d2565-5ea7-4342-a4bc-a69e354fb35e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3c3d2565-5ea7-4342-a4bc-a69e354fb35e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:55:29,256 [235] [DEBUG] [app] Ending request: urn:request:435032b2-4b6b-4fb8-90a0-452004563986 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:435032b2-4b6b-4fb8-90a0-452004563986', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:55:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:29,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:55:29,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
storagereplication stdout | 2025-02-07 04:55:30,014 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 04:55:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:55:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:32.625945+00:00 (in 1.004353 seconds)
namespacegcworker stdout | 2025-02-07 04:55:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:31 UTC)" (scheduled at 2025-02-07 04:55:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:55:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:55:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 31, 621860), True, datetime.datetime(2025, 2, 7, 4, 55, 31, 621860), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:55:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:55:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:55:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:55:31,734 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:55:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:55:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:31.621131+00:00 (in 58.994730 seconds)
namespacegcworker stdout | 2025-02-07 04:55:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:32 UTC)" (scheduled at 2025-02-07 04:55:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:55:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:55:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:55:32,890 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:55:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:55:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:03.919286+00:00 (in 29.999491 seconds)
buildlogsarchiver stdout | 2025-02-07 04:55:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:03 UTC)" (scheduled at 2025-02-07 04:55:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:55:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 55, 33, 920080), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:55:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:55:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:55:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:55:37,180 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:56.713547+00:00 (in 18.997201 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:37 UTC)" (scheduled at 2025-02-07 04:55:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:55:38,290 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:55:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:55:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:48.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 04:55:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:48 UTC)" (scheduled at 2025-02-07 04:55:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:55:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:55:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 38, 732878), True, datetime.datetime(2025, 2, 7, 4, 55, 38, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:55:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:55:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:55:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:55:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:55:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:54.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 04:55:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:10 UTC)" (scheduled at 2025-02-07 04:55:40.424250+00:00)
gcworker stdout | 2025-02-07 04:55:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:55:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903840431, None, 1, 0])
gcworker stdout | 2025-02-07 04:55:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:55:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:55:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:55:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:11.831521+00:00 (in 29.999505 seconds)
autopruneworker stdout | 2025-02-07 04:55:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:11 UTC)" (scheduled at 2025-02-07 04:55:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:55:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900541836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:55:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:55:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:55:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:55:42,171 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:55:42,770 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:55:43,247 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:55:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:55:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:59.225378+00:00 (in 14.999504 seconds)
quotatotalworker stdout | 2025-02-07 04:55:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:59 UTC)" (scheduled at 2025-02-07 04:55:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:55:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:55:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:55:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:55:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:55:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:55:44,235 [236] [DEBUG] [app] Starting request: urn:request:a6768212-c44e-4177-9749-c7490887f2eb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:44,235 [235] [DEBUG] [app] Starting request: urn:request:8dfbc474-9003-4a4a-87a2-be406d2be4e7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:55:44,238 [245] [DEBUG] [app] Starting request: urn:request:5af1e12a-e7fa-4c6c-b906-3275c8fb3f85 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:44,238 [239] [DEBUG] [app] Starting request: urn:request:268af4bd-77e7-41a3-866a-5522c34c90bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:44,239 [245] [DEBUG] [app] Ending request: urn:request:5af1e12a-e7fa-4c6c-b906-3275c8fb3f85 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5af1e12a-e7fa-4c6c-b906-3275c8fb3f85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:55:44,239 [239] [DEBUG] [app] Ending request: urn:request:268af4bd-77e7-41a3-866a-5522c34c90bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:268af4bd-77e7-41a3-866a-5522c34c90bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:55:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:55:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:44,242 [235] [DEBUG] [app] Starting request: urn:request:579ed125-945c-42b5-9193-f30501ce0b48 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:44,242 [238] [DEBUG] [app] Starting request: urn:request:17241def-e8c9-48c3-98f2-1ee9373580e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:44,242 [235] [DEBUG] [app] Ending request: urn:request:579ed125-945c-42b5-9193-f30501ce0b48 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:579ed125-945c-42b5-9193-f30501ce0b48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:55:44,242 [238] [DEBUG] [app] Ending request: urn:request:17241def-e8c9-48c3-98f2-1ee9373580e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17241def-e8c9-48c3-98f2-1ee9373580e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:55:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:55:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:44,254 [235] [DEBUG] [app] Ending request: urn:request:8dfbc474-9003-4a4a-87a2-be406d2be4e7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8dfbc474-9003-4a4a-87a2-be406d2be4e7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:55:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:44,254 [236] [DEBUG] [app] Ending request: urn:request:a6768212-c44e-4177-9749-c7490887f2eb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6768212-c44e-4177-9749-c7490887f2eb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:55:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:55:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
expiredappspecifictokenworker stdout | 2025-02-07 04:55:44,670 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:55:45,530 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:55:45,605 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:55:46,714 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:55:47,193 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:47,197 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-web stdout | 2025-02-07 04:55:47,203 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:55:47,207 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:55:47,209 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:55:47,212 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
globalpromstats stdout | 2025-02-07 04:55:47,625 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:55:47,715 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:55:47,853 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:55:48,216 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:55:48,219 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:55:48,222 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:55:48,226 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:55:48,662 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:55:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:55:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:50.814677+00:00 (in 2.082078 seconds)
notificationworker stdout | 2025-02-07 04:55:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:58 UTC)" (scheduled at 2025-02-07 04:55:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:55:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:55:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 48, 732923), True, datetime.datetime(2025, 2, 7, 4, 55, 48, 732923), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:55:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:55:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:55:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:55:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:55:48,918 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:55:49,042 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:55:49,502 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:55:49,505 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:55:49,509 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:55:49,512 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:55:49,515 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:55:49,518 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:55:49,520 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:55:49,524 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 04:55:49,527 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
buildlogsarchiver stdout | 2025-02-07 04:55:50,575 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:55:50,583 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
notificationworker stdout | 2025-02-07 04:55:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:55:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:55:58.732147+00:00 (in 7.916975 seconds)
notificationworker stdout | 2025-02-07 04:55:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:50 UTC)" (scheduled at 2025-02-07 04:55:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:55:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:55:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:50 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:55:51,035 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 04:55:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:55:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:10.424250+00:00 (in 16.000701 seconds)
gcworker stdout | 2025-02-07 04:55:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:24 UTC)" (scheduled at 2025-02-07 04:55:54.423055+00:00)
gcworker stdout | 2025-02-07 04:55:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:55:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:55:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:55:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:56.118855+00:00 (in 59.999496 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:55:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:56 UTC)" (scheduled at 2025-02-07 04:55:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:55:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:55:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:55:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:37.715878+00:00 (in 41.001844 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:56 UTC)" (scheduled at 2025-02-07 04:55:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:55:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:55:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:55:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:08.732147+00:00 (in 9.999530 seconds)
notificationworker stdout | 2025-02-07 04:55:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:08 UTC)" (scheduled at 2025-02-07 04:55:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:55:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:55:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 55, 58, 732886), True, datetime.datetime(2025, 2, 7, 4, 55, 58, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:55:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:55:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:55:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:55:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:55:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:14.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 04:55:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:14 UTC)" (scheduled at 2025-02-07 04:55:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:55:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:55:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:55:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:55:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:55:59,235 [236] [DEBUG] [app] Starting request: urn:request:c62f8c24-7139-47f0-8a57-3b8018f53a02 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:59,235 [237] [DEBUG] [app] Starting request: urn:request:bd822651-3633-446b-bf52-716982496f49 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:55:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:55:59,238 [239] [DEBUG] [app] Starting request: urn:request:ff0b99b1-f417-4638-b988-d5949bc77655 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:59,238 [246] [DEBUG] [app] Starting request: urn:request:0f44f64c-929c-426a-97f5-101766423124 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:55:59,238 [239] [DEBUG] [app] Ending request: urn:request:ff0b99b1-f417-4638-b988-d5949bc77655 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff0b99b1-f417-4638-b988-d5949bc77655', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:55:59,238 [246] [DEBUG] [app] Ending request: urn:request:0f44f64c-929c-426a-97f5-101766423124 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f44f64c-929c-426a-97f5-101766423124', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:55:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:55:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:55:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:55:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:55:59,241 [237] [DEBUG] [app] Starting request: urn:request:90e6dbcc-e9cc-416e-8d73-e7afd0316e68 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:59,241 [237] [DEBUG] [app] Ending request: urn:request:90e6dbcc-e9cc-416e-8d73-e7afd0316e68 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:90e6dbcc-e9cc-416e-8d73-e7afd0316e68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:55:59,242 [238] [DEBUG] [app] Starting request: urn:request:bcaadcc2-e6d7-4ef9-ae7d-07a1ffd56a20 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:55:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:55:59,242 [238] [DEBUG] [app] Ending request: urn:request:bcaadcc2-e6d7-4ef9-ae7d-07a1ffd56a20 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bcaadcc2-e6d7-4ef9-ae7d-07a1ffd56a20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:55:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:55:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:55:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:55:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:55:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:55:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:55:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:55:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:55:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:55:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:55:59,253 [237] [DEBUG] [app] Ending request: urn:request:bd822651-3633-446b-bf52-716982496f49 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd822651-3633-446b-bf52-716982496f49', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:55:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:55:59,253 [236] [DEBUG] [app] Ending request: urn:request:c62f8c24-7139-47f0-8a57-3b8018f53a02 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c62f8c24-7139-47f0-8a57-3b8018f53a02', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:55:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:55:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:55:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:55:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 04:55:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:55:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:04.936688+00:00 (in 4.998295 seconds)
exportactionlogsworker stdout | 2025-02-07 04:55:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:59 UTC)" (scheduled at 2025-02-07 04:55:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:55:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:55:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:56:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:56:00,050 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
expiredappspecifictokenworker stdout | 2025-02-07 04:56:00,815 [60] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2025-02-07 04:56:00,815 [60] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:56:00.815400+00:00 (in 3599.999479 seconds)
expiredappspecifictokenworker stdout | 2025-02-07 04:56:00,816 [60] [INFO] [apscheduler.executors.default] Running job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2025-02-07 05:56:00 UTC)" (scheduled at 2025-02-07 04:56:00.815400+00:00)
expiredappspecifictokenworker stdout | 2025-02-07 04:56:00,816 [60] [DEBUG] [__main__] Garbage collecting expired app specific tokens with window: 1 day, 0:00:00
expiredappspecifictokenworker stdout | 2025-02-07 04:56:00,816 [60] [DEBUG] [peewee] ('DELETE FROM "appspecificauthtoken" WHERE ("appspecificauthtoken"."expiration" < %s)', [datetime.datetime(2025, 2, 6, 4, 56, 0, 816295)])
expiredappspecifictokenworker stdout | 2025-02-07 04:56:00,822 [60] [DEBUG] [data.database] Disconnecting from database.
expiredappspecifictokenworker stdout | 2025-02-07 04:56:00,822 [60] [INFO] [apscheduler.executors.default] Job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2025-02-07 05:56:00 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:56:01,741 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:56:02,907 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:56:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:56:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:23.420886+00:00 (in 20.001307 seconds)
repositorygcworker stdout | 2025-02-07 04:56:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:03 UTC)" (scheduled at 2025-02-07 04:56:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:56:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:56:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 3, 419855), True, datetime.datetime(2025, 2, 7, 4, 56, 3, 419855), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:56:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:56:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:56:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:56:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:56:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:33.919286+00:00 (in 29.999458 seconds)
buildlogsarchiver stdout | 2025-02-07 04:56:03,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:33 UTC)" (scheduled at 2025-02-07 04:56:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:56:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 56, 3, 920121), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:56:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:56:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:56:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:56:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:56:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:59.937899+00:00 (in 55.000719 seconds)
exportactionlogsworker stdout | 2025-02-07 04:56:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:04 UTC)" (scheduled at 2025-02-07 04:56:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:56:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:56:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 4, 937503), True, datetime.datetime(2025, 2, 7, 4, 56, 4, 937503), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:56:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:56:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:56:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:56:07,210 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:56:08,326 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:56:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:56:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:18.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 04:56:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:18 UTC)" (scheduled at 2025-02-07 04:56:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:56:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:56:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 8, 732934), True, datetime.datetime(2025, 2, 7, 4, 56, 8, 732934), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:56:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:56:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:56:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:56:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:56:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:24.423055+00:00 (in 13.998364 seconds)
gcworker stdout | 2025-02-07 04:56:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:40 UTC)" (scheduled at 2025-02-07 04:56:10.424250+00:00)
gcworker stdout | 2025-02-07 04:56:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:56:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903870431, None, 1, 0])
gcworker stdout | 2025-02-07 04:56:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:56:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:56:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:56:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:41.831521+00:00 (in 29.999516 seconds)
autopruneworker stdout | 2025-02-07 04:56:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:41 UTC)" (scheduled at 2025-02-07 04:56:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:56:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900571836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:56:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:56:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:56:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:56:12,207 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:56:12,786 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:56:13,269 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:56:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:56:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:29.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 04:56:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:29 UTC)" (scheduled at 2025-02-07 04:56:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:56:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:56:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:56:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:56:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:56:14,236 [238] [DEBUG] [app] Starting request: urn:request:736210dc-5771-41fe-9954-06f665aeb967 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:14,236 [235] [DEBUG] [app] Starting request: urn:request:c22819c5-9572-42db-9687-58f305279adb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:56:14,239 [239] [DEBUG] [app] Starting request: urn:request:4c0c114b-5667-4302-a13f-02e019d1ba99 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:14,239 [246] [DEBUG] [app] Starting request: urn:request:a3111a35-69fe-454e-80e7-5887b02b9e8b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:14,239 [239] [DEBUG] [app] Ending request: urn:request:4c0c114b-5667-4302-a13f-02e019d1ba99 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4c0c114b-5667-4302-a13f-02e019d1ba99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:56:14,239 [246] [DEBUG] [app] Ending request: urn:request:a3111a35-69fe-454e-80e7-5887b02b9e8b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3111a35-69fe-454e-80e7-5887b02b9e8b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:56:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:56:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:56:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:14,242 [235] [DEBUG] [app] Starting request: urn:request:ef2e1d90-f34b-42e2-88c7-29fe98ecf92a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:14,242 [235] [DEBUG] [app] Ending request: urn:request:ef2e1d90-f34b-42e2-88c7-29fe98ecf92a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef2e1d90-f34b-42e2-88c7-29fe98ecf92a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:14,242 [237] [DEBUG] [app] Starting request: urn:request:d49a9d88-a98a-4689-9ffd-6c2a2b37bd30 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:14,243 [237] [DEBUG] [app] Ending request: urn:request:d49a9d88-a98a-4689-9ffd-6c2a2b37bd30 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d49a9d88-a98a-4689-9ffd-6c2a2b37bd30', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:56:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:56:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:14,254 [235] [DEBUG] [app] Ending request: urn:request:c22819c5-9572-42db-9687-58f305279adb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c22819c5-9572-42db-9687-58f305279adb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:56:14,254 [238] [DEBUG] [app] Ending request: urn:request:736210dc-5771-41fe-9954-06f665aeb967 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:736210dc-5771-41fe-9954-06f665aeb967', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:56:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:56:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:56:14,678 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:56:15,566 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:56:15,641 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:56:16,724 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:56:17,202 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:56:17,211 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:56:17,214 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:56:17,217 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:56:17,220 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:17,230 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:56:17,661 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:56:17,734 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:56:17,889 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:56:18,224 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:56:18,227 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:56:18,230 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:56:18,261 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:56:18,688 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:56:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:56:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:28.732147+00:00 (in 9.999528 seconds)
notificationworker stdout | 2025-02-07 04:56:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:28 UTC)" (scheduled at 2025-02-07 04:56:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:56:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:56:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 18, 732890), True, datetime.datetime(2025, 2, 7, 4, 56, 18, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:56:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:56:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:56:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:28 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:56:18,937 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:56:19,079 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:56:19,513 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:56:19,518 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:56:19,520 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:56:19,523 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:56:19,526 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:56:19,528 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:56:19,531 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:56:19,535 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:56:19,537 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:56:20,589 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:56:20,617 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:56:21,049 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:56:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:56:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:03.419110+00:00 (in 39.997756 seconds)
repositorygcworker stdout | 2025-02-07 04:56:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:23 UTC)" (scheduled at 2025-02-07 04:56:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:56:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:56:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:56:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:56:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:40.424250+00:00 (in 16.000679 seconds)
gcworker stdout | 2025-02-07 04:56:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:54 UTC)" (scheduled at 2025-02-07 04:56:24.423055+00:00)
gcworker stdout | 2025-02-07 04:56:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:56:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:56:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:56:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:56:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:38.732147+00:00 (in 9.999597 seconds)
notificationworker stdout | 2025-02-07 04:56:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:38 UTC)" (scheduled at 2025-02-07 04:56:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:56:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:56:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 28, 732820), True, datetime.datetime(2025, 2, 7, 4, 56, 28, 732820), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:56:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:56:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:56:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:56:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:56:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:44.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 04:56:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:44 UTC)" (scheduled at 2025-02-07 04:56:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:56:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:56:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:56:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:56:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:56:29,235 [237] [DEBUG] [app] Starting request: urn:request:63b75d35-34cc-430c-8dfc-b2875b362036 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:29,235 [235] [DEBUG] [app] Starting request: urn:request:66678755-a507-4761-8b21-d06e31958da6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:56:29,238 [245] [DEBUG] [app] Starting request: urn:request:26f3a7fb-0046-4308-80ba-1b60dc072467 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:29,238 [243] [DEBUG] [app] Starting request: urn:request:5a54e9d9-e4a9-4162-954c-2cb3197d6c07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:29,238 [245] [DEBUG] [app] Ending request: urn:request:26f3a7fb-0046-4308-80ba-1b60dc072467 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:26f3a7fb-0046-4308-80ba-1b60dc072467', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:56:29,238 [243] [DEBUG] [app] Ending request: urn:request:5a54e9d9-e4a9-4162-954c-2cb3197d6c07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5a54e9d9-e4a9-4162-954c-2cb3197d6c07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:56:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:56:29,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:56:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:29,241 [237] [DEBUG] [app] Starting request: urn:request:19a141f0-1aaf-4056-9134-8144fd3ffd1e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:29,241 [237] [DEBUG] [app] Ending request: urn:request:19a141f0-1aaf-4056-9134-8144fd3ffd1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:19a141f0-1aaf-4056-9134-8144fd3ffd1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:29,241 [236] [DEBUG] [app] Starting request: urn:request:7746e6d5-9fb1-4f95-acc3-ef71adc8dfe5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:29,241 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:29,242 [236] [DEBUG] [app] Ending request: urn:request:7746e6d5-9fb1-4f95-acc3-ef71adc8dfe5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7746e6d5-9fb1-4f95-acc3-ef71adc8dfe5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:56:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:56:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:29,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:29,253 [237] [DEBUG] [app] Ending request: urn:request:63b75d35-34cc-430c-8dfc-b2875b362036 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63b75d35-34cc-430c-8dfc-b2875b362036', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 04:56:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:56:29,254 [235] [DEBUG] [app] Ending request: urn:request:66678755-a507-4761-8b21-d06e31958da6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66678755-a507-4761-8b21-d06e31958da6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:56:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 04:56:30,078 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 04:56:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:56:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:32.625945+00:00 (in 1.004402 seconds)
namespacegcworker stdout | 2025-02-07 04:56:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:31 UTC)" (scheduled at 2025-02-07 04:56:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:56:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:56:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 31, 621747), True, datetime.datetime(2025, 2, 7, 4, 56, 31, 621747), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:56:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:56:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:56:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:56:31,778 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:56:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:56:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:13.624301+00:00 (in 40.997882 seconds)
namespacegcworker stdout | 2025-02-07 04:56:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:32 UTC)" (scheduled at 2025-02-07 04:56:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:56:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:56:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:56:32,925 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:56:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:56:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:03.919286+00:00 (in 29.999503 seconds)
buildlogsarchiver stdout | 2025-02-07 04:56:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:03 UTC)" (scheduled at 2025-02-07 04:56:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:56:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 56, 33, 920035), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:56:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:56:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:56:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:56:37,242 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:56.713547+00:00 (in 18.997166 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:37 UTC)" (scheduled at 2025-02-07 04:56:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:56:38,335 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:56:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:56:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:48.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 04:56:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:48 UTC)" (scheduled at 2025-02-07 04:56:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:56:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:56:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 38, 732884), True, datetime.datetime(2025, 2, 7, 4, 56, 38, 732884), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:56:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:56:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:56:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:56:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:56:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:54.423055+00:00 (in 13.998323 seconds)
gcworker stdout | 2025-02-07 04:56:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:10 UTC)" (scheduled at 2025-02-07 04:56:40.424250+00:00)
gcworker stdout | 2025-02-07 04:56:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:56:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903900431, None, 1, 0])
gcworker stdout | 2025-02-07 04:56:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:56:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:56:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:56:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:11.831521+00:00 (in 29.999542 seconds)
autopruneworker stdout | 2025-02-07 04:56:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:11 UTC)" (scheduled at 2025-02-07 04:56:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:56:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900601836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:56:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:56:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:56:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:56:42,217 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:56:42,796 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:56:43,305 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:56:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:56:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:59.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 04:56:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:59 UTC)" (scheduled at 2025-02-07 04:56:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:56:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:56:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:56:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:56:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:56:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:56:44,235 [238] [DEBUG] [app] Starting request: urn:request:07d798e3-958d-40a0-962c-b48658a108c6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:44,235 [237] [DEBUG] [app] Starting request: urn:request:6272af81-b0b5-410a-b769-5df8e4d5e4a9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:56:44,238 [239] [DEBUG] [app] Starting request: urn:request:45e3e158-03df-4bba-860f-6ce945b9a63f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:44,238 [244] [DEBUG] [app] Starting request: urn:request:0f16670e-2183-4443-8a90-2579c00af7e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:44,239 [239] [DEBUG] [app] Ending request: urn:request:45e3e158-03df-4bba-860f-6ce945b9a63f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:45e3e158-03df-4bba-860f-6ce945b9a63f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:56:44,239 [244] [DEBUG] [app] Ending request: urn:request:0f16670e-2183-4443-8a90-2579c00af7e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f16670e-2183-4443-8a90-2579c00af7e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:56:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:56:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:56:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:44,242 [238] [DEBUG] [app] Starting request: urn:request:7f27e04a-e5a1-4ace-be82-879c0d48e059 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:44,242 [236] [DEBUG] [app] Starting request: urn:request:b430fe6d-c82a-4d2e-a037-3875a6483354 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:44,242 [238] [DEBUG] [app] Ending request: urn:request:7f27e04a-e5a1-4ace-be82-879c0d48e059 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f27e04a-e5a1-4ace-be82-879c0d48e059', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:44,242 [236] [DEBUG] [app] Ending request: urn:request:b430fe6d-c82a-4d2e-a037-3875a6483354 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b430fe6d-c82a-4d2e-a037-3875a6483354', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:56:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:56:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:44,253 [237] [DEBUG] [app] Ending request: urn:request:6272af81-b0b5-410a-b769-5df8e4d5e4a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6272af81-b0b5-410a-b769-5df8e4d5e4a9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:56:44,254 [238] [DEBUG] [app] Ending request: urn:request:07d798e3-958d-40a0-962c-b48658a108c6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:07d798e3-958d-40a0-962c-b48658a108c6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:56:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:56:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
expiredappspecifictokenworker stdout | 2025-02-07 04:56:44,715 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:56:45,578 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:56:45,662 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:56:46,760 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:56:47,209 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:56:47,220 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:56:47,223 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:56:47,226 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:56:47,228 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:47,267 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:56:47,668 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:56:47,742 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:56:47,899 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:56:48,233 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:56:48,236 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:56:48,238 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:56:48,281 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 04:56:48,715 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 04:56:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:56:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:50.814677+00:00 (in 2.082130 seconds)
notificationworker stdout | 2025-02-07 04:56:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:58 UTC)" (scheduled at 2025-02-07 04:56:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:56:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:56:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 48, 732825), True, datetime.datetime(2025, 2, 7, 4, 56, 48, 732825), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:56:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:56:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:56:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:56:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 04:56:48,945 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:56:49,098 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:56:49,523 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:56:49,526 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:56:49,531 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:56:49,535 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:56:49,539 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:56:49,543 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:56:49,546 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:56:49,549 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:56:49,552 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:56:50,625 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:56:50,652 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
notificationworker stdout | 2025-02-07 04:56:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:56:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:56:58.732147+00:00 (in 7.916998 seconds)
notificationworker stdout | 2025-02-07 04:56:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:50 UTC)" (scheduled at 2025-02-07 04:56:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:56:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:56:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:50 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:56:51,079 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 04:56:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:56:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:10.424250+00:00 (in 16.000745 seconds)
gcworker stdout | 2025-02-07 04:56:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:24 UTC)" (scheduled at 2025-02-07 04:56:54.423055+00:00)
gcworker stdout | 2025-02-07 04:56:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:56:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:56:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:56:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:56.118855+00:00 (in 59.999481 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:56:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:56 UTC)" (scheduled at 2025-02-07 04:56:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:56:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:56:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:56:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:37.715878+00:00 (in 41.001861 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:56 UTC)" (scheduled at 2025-02-07 04:56:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:56:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:56:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:56:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:08.732147+00:00 (in 9.999533 seconds)
notificationworker stdout | 2025-02-07 04:56:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:08 UTC)" (scheduled at 2025-02-07 04:56:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:56:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:56:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 56, 58, 732953), True, datetime.datetime(2025, 2, 7, 4, 56, 58, 732953), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:56:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:56:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:56:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:56:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:56:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:14.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 04:56:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:14 UTC)" (scheduled at 2025-02-07 04:56:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:56:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:56:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:56:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:56:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:56:59,236 [236] [DEBUG] [app] Starting request: urn:request:bef06e1c-2401-486d-a069-7cfa4f593263 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:59,236 [235] [DEBUG] [app] Starting request: urn:request:57109946-57bd-4384-a187-76b9da15dcb7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:56:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:59,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:56:59,240 [243] [DEBUG] [app] Starting request: urn:request:a47fd2e2-ef08-4883-bf17-a8571bb3eebf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:59,240 [246] [DEBUG] [app] Starting request: urn:request:bc876994-8325-451d-baf9-dff290227b1b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:56:59,240 [246] [DEBUG] [app] Ending request: urn:request:bc876994-8325-451d-baf9-dff290227b1b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bc876994-8325-451d-baf9-dff290227b1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:56:59,240 [243] [DEBUG] [app] Ending request: urn:request:a47fd2e2-ef08-4883-bf17-a8571bb3eebf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a47fd2e2-ef08-4883-bf17-a8571bb3eebf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:56:59,241 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:56:59,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:56:59,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:59,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:59,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:56:59,244 [236] [DEBUG] [app] Starting request: urn:request:acf402b7-8ff1-4197-8484-396ba03a5f4b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:59,244 [236] [DEBUG] [app] Ending request: urn:request:acf402b7-8ff1-4197-8484-396ba03a5f4b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:acf402b7-8ff1-4197-8484-396ba03a5f4b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:59,244 [238] [DEBUG] [app] Starting request: urn:request:044f63a5-e9ca-46d7-a197-353c8bd25c10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:56:59,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:59,245 [238] [DEBUG] [app] Ending request: urn:request:044f63a5-e9ca-46d7-a197-353c8bd25c10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:044f63a5-e9ca-46d7-a197-353c8bd25c10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:56:59,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:59,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:56:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:56:59,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:56:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:56:59,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:56:59,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:56:59,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:56:59,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:59,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:59,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:56:59,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:56:59,254 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:59,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:56:59,255 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:59,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:56:59,257 [235] [DEBUG] [app] Ending request: urn:request:57109946-57bd-4384-a187-76b9da15dcb7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:57109946-57bd-4384-a187-76b9da15dcb7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:59,257 [236] [DEBUG] [app] Ending request: urn:request:bef06e1c-2401-486d-a069-7cfa4f593263 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bef06e1c-2401-486d-a069-7cfa4f593263', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:56:59,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:56:59,257 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:56:59,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:56:59,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:56:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:56:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
exportactionlogsworker stdout | 2025-02-07 04:56:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:56:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:04.936688+00:00 (in 4.998295 seconds)
exportactionlogsworker stdout | 2025-02-07 04:56:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:59 UTC)" (scheduled at 2025-02-07 04:56:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:56:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:56:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:57:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:57:00,086 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:57:01,794 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:57:02,962 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:57:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:57:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:23.420886+00:00 (in 20.001284 seconds)
repositorygcworker stdout | 2025-02-07 04:57:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:03 UTC)" (scheduled at 2025-02-07 04:57:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:57:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:57:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 3, 419882), True, datetime.datetime(2025, 2, 7, 4, 57, 3, 419882), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:57:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:57:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:57:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:57:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:57:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:33.919286+00:00 (in 29.999488 seconds)
buildlogsarchiver stdout | 2025-02-07 04:57:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:33 UTC)" (scheduled at 2025-02-07 04:57:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:57:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 57, 3, 920053), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:57:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:57:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:57:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:57:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:57:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:59.937899+00:00 (in 55.000756 seconds)
exportactionlogsworker stdout | 2025-02-07 04:57:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:04 UTC)" (scheduled at 2025-02-07 04:57:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:57:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:57:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 4, 937387), True, datetime.datetime(2025, 2, 7, 4, 57, 4, 937387), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:57:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:57:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:57:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:57:07,278 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:57:08,353 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:57:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:18.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 04:57:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:18 UTC)" (scheduled at 2025-02-07 04:57:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:57:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:57:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 8, 732911), True, datetime.datetime(2025, 2, 7, 4, 57, 8, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:57:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:57:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:57:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:57:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:57:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:24.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 04:57:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:40 UTC)" (scheduled at 2025-02-07 04:57:10.424250+00:00)
gcworker stdout | 2025-02-07 04:57:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:57:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903930431, None, 1, 0])
gcworker stdout | 2025-02-07 04:57:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:57:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:57:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:57:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 04:57:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:41 UTC)" (scheduled at 2025-02-07 04:57:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:57:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900631836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:57:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:57:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:57:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:57:12,237 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:57:12,831 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:57:13,342 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
namespacegcworker stdout | 2025-02-07 04:57:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:57:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:31.621131+00:00 (in 17.996364 seconds)
namespacegcworker stdout | 2025-02-07 04:57:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:02:13 UTC)" (scheduled at 2025-02-07 04:57:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 04:57:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 57, 13, 624963), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:57:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 57, 13, 624963), True, datetime.datetime(2025, 2, 7, 4, 57, 13, 624963), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:57:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 57, 13, 624963), True, datetime.datetime(2025, 2, 7, 4, 57, 13, 624963), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 4, 57, 13, 624963), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 04:57:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:57:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:02:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:57:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:57:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:29.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 04:57:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:29 UTC)" (scheduled at 2025-02-07 04:57:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:57:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:57:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:57:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:57:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:57:14,235 [235] [DEBUG] [app] Starting request: urn:request:cab4d503-0107-43c4-9050-30ae12fbd6fa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:14,235 [238] [DEBUG] [app] Starting request: urn:request:28e4a471-2e2a-4147-8367-9193f9c50369 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:57:14,238 [245] [DEBUG] [app] Starting request: urn:request:c4d5363e-e923-4c48-8363-f99307c78993 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:14,238 [244] [DEBUG] [app] Starting request: urn:request:88ba3e33-0256-4aaa-9299-b9a546db490a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:14,238 [244] [DEBUG] [app] Ending request: urn:request:88ba3e33-0256-4aaa-9299-b9a546db490a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:88ba3e33-0256-4aaa-9299-b9a546db490a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:14,239 [245] [DEBUG] [app] Ending request: urn:request:c4d5363e-e923-4c48-8363-f99307c78993 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c4d5363e-e923-4c48-8363-f99307c78993', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:57:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:14,242 [237] [DEBUG] [app] Starting request: urn:request:594d8e0e-c4b1-41d0-bdc9-b727ce155ea4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:14,242 [236] [DEBUG] [app] Starting request: urn:request:8b9846ea-02ff-4184-96d8-5aed1572c015 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:14,243 [236] [DEBUG] [app] Ending request: urn:request:8b9846ea-02ff-4184-96d8-5aed1572c015 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b9846ea-02ff-4184-96d8-5aed1572c015', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:57:14,243 [237] [DEBUG] [app] Ending request: urn:request:594d8e0e-c4b1-41d0-bdc9-b727ce155ea4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:594d8e0e-c4b1-41d0-bdc9-b727ce155ea4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:57:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:57:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:14,254 [238] [DEBUG] [app] Ending request: urn:request:28e4a471-2e2a-4147-8367-9193f9c50369 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28e4a471-2e2a-4147-8367-9193f9c50369', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:57:14,254 [235] [DEBUG] [app] Ending request: urn:request:cab4d503-0107-43c4-9050-30ae12fbd6fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cab4d503-0107-43c4-9050-30ae12fbd6fa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:57:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:57:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:57:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:57:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:57:14,751 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:57:15,614 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:57:15,671 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:57:16,779 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:57:17,221 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:57:17,231 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:57:17,235 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:57:17,238 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:57:17,241 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:17,275 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:57:17,690 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:57:17,778 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:57:17,936 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:57:18,241 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:57:18,245 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:57:18,248 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:57:18,313 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 04:57:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:28.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 04:57:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:28 UTC)" (scheduled at 2025-02-07 04:57:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:57:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:57:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 18, 732875), True, datetime.datetime(2025, 2, 7, 4, 57, 18, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:57:18,744 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:57:18,744 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:57:18,744 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:57:18,751 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:57:18,977 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:57:19,129 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:57:19,532 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:57:19,536 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:57:19,542 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:57:19,546 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:57:19,549 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:57:19,553 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:57:19,556 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:57:19,560 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:57:19,562 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:57:20,640 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:57:20,688 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:57:21,115 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:57:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:57:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:03.419110+00:00 (in 39.997730 seconds)
repositorygcworker stdout | 2025-02-07 04:57:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:23 UTC)" (scheduled at 2025-02-07 04:57:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:57:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:57:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:57:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:57:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:40.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 04:57:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:54 UTC)" (scheduled at 2025-02-07 04:57:24.423055+00:00)
gcworker stdout | 2025-02-07 04:57:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:57:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:57:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:57:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:33.814166+00:00 (in 5.081577 seconds)
notificationworker stdout | 2025-02-07 04:57:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:38 UTC)" (scheduled at 2025-02-07 04:57:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:57:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:57:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 28, 732865), True, datetime.datetime(2025, 2, 7, 4, 57, 28, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:57:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:57:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:57:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:57:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:57:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:44.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 04:57:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:44 UTC)" (scheduled at 2025-02-07 04:57:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:57:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:57:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:57:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:57:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:57:29,235 [238] [DEBUG] [app] Starting request: urn:request:7f9d4b31-139f-48b7-8cab-5990534e4b02 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:29,235 [235] [DEBUG] [app] Starting request: urn:request:7f41ce75-d45f-4766-88b7-8a22b1845586 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:57:29,238 [239] [DEBUG] [app] Starting request: urn:request:9e2e2e72-d141-4f3b-b2a8-40dddc770097 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:29,238 [246] [DEBUG] [app] Starting request: urn:request:c004fc1c-16ca-469d-9dc0-6181be7ca377 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:29,239 [239] [DEBUG] [app] Ending request: urn:request:9e2e2e72-d141-4f3b-b2a8-40dddc770097 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e2e2e72-d141-4f3b-b2a8-40dddc770097', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:29,239 [246] [DEBUG] [app] Ending request: urn:request:c004fc1c-16ca-469d-9dc0-6181be7ca377 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c004fc1c-16ca-469d-9dc0-6181be7ca377', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:57:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:29,242 [237] [DEBUG] [app] Starting request: urn:request:9020afd9-8481-413c-bbd4-adc896691594 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:29,242 [236] [DEBUG] [app] Starting request: urn:request:85617316-fcd8-4f18-811b-911acfc50069 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:29,243 [237] [DEBUG] [app] Ending request: urn:request:9020afd9-8481-413c-bbd4-adc896691594 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9020afd9-8481-413c-bbd4-adc896691594', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:57:29,243 [236] [DEBUG] [app] Ending request: urn:request:85617316-fcd8-4f18-811b-911acfc50069 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:85617316-fcd8-4f18-811b-911acfc50069', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:57:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:29,254 [235] [DEBUG] [app] Ending request: urn:request:7f41ce75-d45f-4766-88b7-8a22b1845586 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f41ce75-d45f-4766-88b7-8a22b1845586', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:57:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:57:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:57:29,254 [238] [DEBUG] [app] Ending request: urn:request:7f9d4b31-139f-48b7-8cab-5990534e4b02 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f9d4b31-139f-48b7-8cab-5990534e4b02', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:57:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:57:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 04:57:30,099 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 04:57:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:57:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:32.625945+00:00 (in 1.004351 seconds)
namespacegcworker stdout | 2025-02-07 04:57:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:31 UTC)" (scheduled at 2025-02-07 04:57:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:57:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:57:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 31, 621806), True, datetime.datetime(2025, 2, 7, 4, 57, 31, 621806), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:57:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:57:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:57:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:57:31,822 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:57:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:57:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:31.621131+00:00 (in 58.994708 seconds)
namespacegcworker stdout | 2025-02-07 04:57:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:32 UTC)" (scheduled at 2025-02-07 04:57:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:57:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:57:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:57:32,998 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 04:57:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:38.732147+00:00 (in 4.917501 seconds)
notificationworker stdout | 2025-02-07 04:57:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:02:33 UTC)" (scheduled at 2025-02-07 04:57:33.814166+00:00)
notificationworker stdout | 2025-02-07 04:57:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 57, 33, 814915), 'notification/%'])
notificationworker stdout | 2025-02-07 04:57:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 57, 33, 814915), True, datetime.datetime(2025, 2, 7, 4, 57, 33, 814915), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 04:57:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 57, 33, 814915), True, datetime.datetime(2025, 2, 7, 4, 57, 33, 814915), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 4, 57, 33, 814915), 'notification/%'])
notificationworker stdout | 2025-02-07 04:57:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:57:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:02:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:57:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:57:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:03.919286+00:00 (in 29.999483 seconds)
buildlogsarchiver stdout | 2025-02-07 04:57:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:03 UTC)" (scheduled at 2025-02-07 04:57:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:57:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 57, 33, 920095), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:57:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:57:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:57:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:57:37,314 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:56.713547+00:00 (in 18.997215 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:37 UTC)" (scheduled at 2025-02-07 04:57:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:57:38,389 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:57:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:48.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 04:57:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:48 UTC)" (scheduled at 2025-02-07 04:57:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:57:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:57:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 38, 732860), True, datetime.datetime(2025, 2, 7, 4, 57, 38, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:57:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:57:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:57:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:57:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:57:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:54.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 04:57:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:10 UTC)" (scheduled at 2025-02-07 04:57:40.424250+00:00)
gcworker stdout | 2025-02-07 04:57:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:57:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903960431, None, 1, 0])
gcworker stdout | 2025-02-07 04:57:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:57:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:57:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:57:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:11.831521+00:00 (in 29.999553 seconds)
autopruneworker stdout | 2025-02-07 04:57:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:11 UTC)" (scheduled at 2025-02-07 04:57:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:57:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900661836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:57:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:57:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:57:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:57:42,273 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:57:42,841 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:57:43,370 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:57:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:57:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:59.225378+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2025-02-07 04:57:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:59 UTC)" (scheduled at 2025-02-07 04:57:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:57:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:57:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:57:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:57:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:57:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:57:44,235 [235] [DEBUG] [app] Starting request: urn:request:6a258c36-1f48-4dbe-a191-c655dad93349 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:44,235 [238] [DEBUG] [app] Starting request: urn:request:631fa921-b7cf-4ff3-8f60-d31e81a1268d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:57:44,238 [244] [DEBUG] [app] Starting request: urn:request:02ed876c-387b-41ce-846f-ac266964691e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:44,238 [239] [DEBUG] [app] Starting request: urn:request:29e790cf-7f31-4dfc-af8d-fa45de473c86 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:44,239 [244] [DEBUG] [app] Ending request: urn:request:02ed876c-387b-41ce-846f-ac266964691e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02ed876c-387b-41ce-846f-ac266964691e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:44,239 [239] [DEBUG] [app] Ending request: urn:request:29e790cf-7f31-4dfc-af8d-fa45de473c86 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:29e790cf-7f31-4dfc-af8d-fa45de473c86', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:57:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:44,242 [237] [DEBUG] [app] Starting request: urn:request:b130d8d1-2021-46b5-a505-9394b4c83301 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:44,242 [237] [DEBUG] [app] Ending request: urn:request:b130d8d1-2021-46b5-a505-9394b4c83301 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b130d8d1-2021-46b5-a505-9394b4c83301', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:57:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:44,242 [235] [DEBUG] [app] Starting request: urn:request:544c5694-ff8a-473d-95e1-0448abc6eb71 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:44,242 [235] [DEBUG] [app] Ending request: urn:request:544c5694-ff8a-473d-95e1-0448abc6eb71 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:544c5694-ff8a-473d-95e1-0448abc6eb71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:57:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:44,254 [235] [DEBUG] [app] Ending request: urn:request:6a258c36-1f48-4dbe-a191-c655dad93349 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a258c36-1f48-4dbe-a191-c655dad93349', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:57:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:57:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:57:44,254 [238] [DEBUG] [app] Ending request: urn:request:631fa921-b7cf-4ff3-8f60-d31e81a1268d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:631fa921-b7cf-4ff3-8f60-d31e81a1268d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:57:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:57:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:57:44,757 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:57:45,650 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:57:45,681 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:57:46,814 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:57:47,228 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:57:47,240 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:57:47,246 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:57:47,249 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:57:47,252 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:47,299 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:57:47,726 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:57:47,810 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:57:47,968 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:57:48,250 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:57:48,253 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:57:48,256 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:57:48,349 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 04:57:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:50.814677+00:00 (in 2.082103 seconds)
notificationworker stdout | 2025-02-07 04:57:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:58 UTC)" (scheduled at 2025-02-07 04:57:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:57:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:57:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 48, 732864), True, datetime.datetime(2025, 2, 7, 4, 57, 48, 732864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:57:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:57:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:57:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:57:58 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:57:48,758 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:57:48,986 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:57:49,166 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:57:49,540 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:57:49,547 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:57:49,552 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:57:49,555 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:57:49,559 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:57:49,562 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:57:49,565 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:57:49,570 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:57:49,573 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:57:50,658 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:57:50,725 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
notificationworker stdout | 2025-02-07 04:57:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:57:58.732147+00:00 (in 7.916961 seconds)
notificationworker stdout | 2025-02-07 04:57:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:50 UTC)" (scheduled at 2025-02-07 04:57:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:57:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:57:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:50 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:57:51,140 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 04:57:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:57:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:10.424250+00:00 (in 16.000744 seconds)
gcworker stdout | 2025-02-07 04:57:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:24 UTC)" (scheduled at 2025-02-07 04:57:54.423055+00:00)
gcworker stdout | 2025-02-07 04:57:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:57:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:57:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:57:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:56.118855+00:00 (in 59.999442 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:57:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:56 UTC)" (scheduled at 2025-02-07 04:57:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:57:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:57:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:57:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:37.715878+00:00 (in 41.001838 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:56 UTC)" (scheduled at 2025-02-07 04:57:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:57:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:57:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:57:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:08.732147+00:00 (in 9.999486 seconds)
notificationworker stdout | 2025-02-07 04:57:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:08 UTC)" (scheduled at 2025-02-07 04:57:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:57:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:57:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 57, 58, 732962), True, datetime.datetime(2025, 2, 7, 4, 57, 58, 732962), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:57:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:57:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:57:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:57:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:57:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:14.225378+00:00 (in 14.999520 seconds)
quotatotalworker stdout | 2025-02-07 04:57:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:14 UTC)" (scheduled at 2025-02-07 04:57:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:57:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:57:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:57:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:57:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:57:59,235 [237] [DEBUG] [app] Starting request: urn:request:14a713e5-cdc7-4f17-a330-510d2c2a91ca (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:59,235 [236] [DEBUG] [app] Starting request: urn:request:a822d5c5-1c25-4913-8347-807a6d4d0154 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:57:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:57:59,238 [239] [DEBUG] [app] Starting request: urn:request:b880d957-93eb-4cee-ac06-1aa3eea039f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:59,239 [246] [DEBUG] [app] Starting request: urn:request:37b46584-97f2-495b-9f0f-4be0919d8705 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:57:59,239 [239] [DEBUG] [app] Ending request: urn:request:b880d957-93eb-4cee-ac06-1aa3eea039f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b880d957-93eb-4cee-ac06-1aa3eea039f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:59,239 [246] [DEBUG] [app] Ending request: urn:request:37b46584-97f2-495b-9f0f-4be0919d8705 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:37b46584-97f2-495b-9f0f-4be0919d8705', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:57:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:57:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:57:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:59,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:57:59,242 [237] [DEBUG] [app] Starting request: urn:request:1477bbae-2f9a-4cc5-ab45-dd6560fe95c6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:59,242 [237] [DEBUG] [app] Ending request: urn:request:1477bbae-2f9a-4cc5-ab45-dd6560fe95c6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1477bbae-2f9a-4cc5-ab45-dd6560fe95c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:57:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:57:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:57:59,243 [235] [DEBUG] [app] Starting request: urn:request:11909baa-03f5-41b8-aff9-9578f0e63b57 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:57:59,243 [235] [DEBUG] [app] Ending request: urn:request:11909baa-03f5-41b8-aff9-9578f0e63b57 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11909baa-03f5-41b8-aff9-9578f0e63b57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:57:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:59,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:57:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:57:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:57:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:57:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:57:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:57:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:57:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:57:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:57:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:57:59,254 [237] [DEBUG] [app] Ending request: urn:request:14a713e5-cdc7-4f17-a330-510d2c2a91ca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14a713e5-cdc7-4f17-a330-510d2c2a91ca', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:57:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:57:59,255 [236] [DEBUG] [app] Ending request: urn:request:a822d5c5-1c25-4913-8347-807a6d4d0154 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a822d5c5-1c25-4913-8347-807a6d4d0154', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:57:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:57:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:57:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:57:59,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:57:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:57:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:57:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:04.936688+00:00 (in 4.998285 seconds)
exportactionlogsworker stdout | 2025-02-07 04:57:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:59 UTC)" (scheduled at 2025-02-07 04:57:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:57:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:57:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:58:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:58:00,120 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:58:01,858 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:58:03,026 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:58:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:58:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:23.420886+00:00 (in 20.001322 seconds)
repositorygcworker stdout | 2025-02-07 04:58:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:03 UTC)" (scheduled at 2025-02-07 04:58:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:58:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:58:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 3, 419836), True, datetime.datetime(2025, 2, 7, 4, 58, 3, 419836), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:58:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:58:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:58:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:58:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:58:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:33.919286+00:00 (in 29.999542 seconds)
buildlogsarchiver stdout | 2025-02-07 04:58:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:33 UTC)" (scheduled at 2025-02-07 04:58:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:58:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 58, 3, 919995), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:58:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:58:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:58:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:58:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:58:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:59.937899+00:00 (in 55.000752 seconds)
exportactionlogsworker stdout | 2025-02-07 04:58:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:04 UTC)" (scheduled at 2025-02-07 04:58:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:58:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:58:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 4, 937481), True, datetime.datetime(2025, 2, 7, 4, 58, 4, 937481), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:58:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:58:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:58:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:58:07,350 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:58:08,426 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:58:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:58:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:18.732147+00:00 (in 9.999465 seconds)
notificationworker stdout | 2025-02-07 04:58:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:18 UTC)" (scheduled at 2025-02-07 04:58:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:58:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:58:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 8, 732998), True, datetime.datetime(2025, 2, 7, 4, 58, 8, 732998), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:58:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:58:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:58:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:58:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:58:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:24.423055+00:00 (in 13.998364 seconds)
gcworker stdout | 2025-02-07 04:58:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:40 UTC)" (scheduled at 2025-02-07 04:58:10.424250+00:00)
gcworker stdout | 2025-02-07 04:58:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:58:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738903990430, None, 1, 0])
gcworker stdout | 2025-02-07 04:58:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:58:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:58:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:58:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:41.831521+00:00 (in 29.999569 seconds)
autopruneworker stdout | 2025-02-07 04:58:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:41 UTC)" (scheduled at 2025-02-07 04:58:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:58:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900691836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:58:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:58:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:58:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:58:12,304 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:58:12,872 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:58:13,401 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:58:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:58:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:29.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 04:58:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:29 UTC)" (scheduled at 2025-02-07 04:58:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:58:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:58:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:58:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:58:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:58:14,235 [235] [DEBUG] [app] Starting request: urn:request:d20e6234-a1aa-4205-9e70-27d84622dc4f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:14,235 [236] [DEBUG] [app] Starting request: urn:request:67f4270d-9f93-4a9d-bb5e-8ba44be1714a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:58:14,238 [245] [DEBUG] [app] Starting request: urn:request:edbc3bd1-a96c-4fd8-835e-10a5a218046d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:14,238 [245] [DEBUG] [app] Ending request: urn:request:edbc3bd1-a96c-4fd8-835e-10a5a218046d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:edbc3bd1-a96c-4fd8-835e-10a5a218046d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:58:14,238 [246] [DEBUG] [app] Starting request: urn:request:48a47452-11e6-45cf-8251-1d78922b8355 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:14,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:58:14,238 [246] [DEBUG] [app] Ending request: urn:request:48a47452-11e6-45cf-8251-1d78922b8355 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48a47452-11e6-45cf-8251-1d78922b8355', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:58:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:58:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:58:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:14,241 [235] [DEBUG] [app] Starting request: urn:request:4d9e8e4d-6114-4b9a-8518-9c2e4676be3e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:14,241 [235] [DEBUG] [app] Ending request: urn:request:4d9e8e4d-6114-4b9a-8518-9c2e4676be3e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d9e8e4d-6114-4b9a-8518-9c2e4676be3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:58:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 04:58:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:14,242 [238] [DEBUG] [app] Starting request: urn:request:7311005e-df62-402e-8b55-7b8394c85004 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:14,242 [238] [DEBUG] [app] Ending request: urn:request:7311005e-df62-402e-8b55-7b8394c85004 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7311005e-df62-402e-8b55-7b8394c85004', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:58:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:58:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:14,253 [235] [DEBUG] [app] Ending request: urn:request:d20e6234-a1aa-4205-9e70-27d84622dc4f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d20e6234-a1aa-4205-9e70-27d84622dc4f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:58:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:58:14,254 [236] [DEBUG] [app] Ending request: urn:request:67f4270d-9f93-4a9d-bb5e-8ba44be1714a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:67f4270d-9f93-4a9d-bb5e-8ba44be1714a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:58:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
expiredappspecifictokenworker stdout | 2025-02-07 04:58:14,794 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:58:15,687 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:58:15,715 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:58:16,851 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:58:17,236 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:58:17,247 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:58:17,252 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:58:17,255 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:58:17,259 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:17,321 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:58:17,751 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:58:17,831 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:58:18,002 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:58:18,258 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:58:18,261 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:58:18,264 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:58:18,385 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 04:58:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:58:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:28.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 04:58:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:28 UTC)" (scheduled at 2025-02-07 04:58:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:58:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:58:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 18, 732899), True, datetime.datetime(2025, 2, 7, 4, 58, 18, 732899), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:58:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:58:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:58:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:58:18,773 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:58:19,017 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:58:19,201 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:58:19,546 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:58:19,558 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:58:19,562 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:58:19,564 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:58:19,568 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:58:19,571 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:58:19,573 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:58:19,580 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:58:19,583 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:58:20,689 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:58:20,761 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:58:21,176 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:58:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:58:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:32.420707+00:00 (in 8.999353 seconds)
repositorygcworker stdout | 2025-02-07 04:58:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:23 UTC)" (scheduled at 2025-02-07 04:58:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:58:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:58:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:58:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:58:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:40.424250+00:00 (in 16.000686 seconds)
gcworker stdout | 2025-02-07 04:58:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:54 UTC)" (scheduled at 2025-02-07 04:58:24.423055+00:00)
gcworker stdout | 2025-02-07 04:58:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:58:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:58:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:58:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:58:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:38.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 04:58:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:38 UTC)" (scheduled at 2025-02-07 04:58:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:58:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:58:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 28, 732858), True, datetime.datetime(2025, 2, 7, 4, 58, 28, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:58:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:58:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:58:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:58:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:58:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:44.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 04:58:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:44 UTC)" (scheduled at 2025-02-07 04:58:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:58:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:58:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:58:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:58:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:58:29,235 [235] [DEBUG] [app] Starting request: urn:request:7e5b286e-f23a-4b75-9c40-b54002d98a53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:29,235 [238] [DEBUG] [app] Starting request: urn:request:c9712521-473b-4fe4-8e13-1c0d5e0beebd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:58:29,238 [246] [DEBUG] [app] Starting request: urn:request:357ca4a8-5643-40db-98ea-15b2955477e4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:29,238 [239] [DEBUG] [app] Starting request: urn:request:0553dcd6-9fef-4ca3-a0ef-60945b3af017 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:29,238 [246] [DEBUG] [app] Ending request: urn:request:357ca4a8-5643-40db-98ea-15b2955477e4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:357ca4a8-5643-40db-98ea-15b2955477e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:58:29,239 [239] [DEBUG] [app] Ending request: urn:request:0553dcd6-9fef-4ca3-a0ef-60945b3af017 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0553dcd6-9fef-4ca3-a0ef-60945b3af017', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:58:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:58:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:58:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:29,241 [236] [DEBUG] [app] Starting request: urn:request:f966f1dc-84f5-47d1-b65b-c402bcad4913 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:29,242 [236] [DEBUG] [app] Ending request: urn:request:f966f1dc-84f5-47d1-b65b-c402bcad4913 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f966f1dc-84f5-47d1-b65b-c402bcad4913', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:58:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:58:29,242 [235] [DEBUG] [app] Starting request: urn:request:1c28c81f-a2a7-4ac2-ac70-88b94bb99920 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:29,242 [235] [DEBUG] [app] Ending request: urn:request:1c28c81f-a2a7-4ac2-ac70-88b94bb99920 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1c28c81f-a2a7-4ac2-ac70-88b94bb99920', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:58:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:58:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:29,253 [238] [DEBUG] [app] Ending request: urn:request:c9712521-473b-4fe4-8e13-1c0d5e0beebd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9712521-473b-4fe4-8e13-1c0d5e0beebd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:58:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:58:29,254 [235] [DEBUG] [app] Ending request: urn:request:7e5b286e-f23a-4b75-9c40-b54002d98a53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e5b286e-f23a-4b75-9c40-b54002d98a53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:58:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 04:58:30,156 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 04:58:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:58:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:32.625945+00:00 (in 1.004302 seconds)
namespacegcworker stdout | 2025-02-07 04:58:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:31 UTC)" (scheduled at 2025-02-07 04:58:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:58:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:58:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 31, 621965), True, datetime.datetime(2025, 2, 7, 4, 58, 31, 621965), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:58:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:58:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:58:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:58:31,894 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 04:58:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:58:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:03.419110+00:00 (in 30.997883 seconds)
repositorygcworker stdout | 2025-02-07 04:58:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:03:32 UTC)" (scheduled at 2025-02-07 04:58:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 04:58:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 58, 32, 421527), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:58:32,427 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 58, 32, 421527), True, datetime.datetime(2025, 2, 7, 4, 58, 32, 421527), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:58:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 58, 32, 421527), True, datetime.datetime(2025, 2, 7, 4, 58, 32, 421527), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 4, 58, 32, 421527), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 04:58:32,430 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:58:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:03:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:58:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:58:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:31.621131+00:00 (in 58.994662 seconds)
namespacegcworker stdout | 2025-02-07 04:58:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:32 UTC)" (scheduled at 2025-02-07 04:58:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:58:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:58:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:58:33,062 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:58:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:58:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:03.919286+00:00 (in 29.999539 seconds)
buildlogsarchiver stdout | 2025-02-07 04:58:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:03 UTC)" (scheduled at 2025-02-07 04:58:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:58:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 58, 33, 919992), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:58:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:58:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:58:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:58:37,386 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:56.713547+00:00 (in 18.997179 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:37 UTC)" (scheduled at 2025-02-07 04:58:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:58:38,444 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:58:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:58:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:48.732147+00:00 (in 9.999578 seconds)
notificationworker stdout | 2025-02-07 04:58:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:48 UTC)" (scheduled at 2025-02-07 04:58:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:58:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:58:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 38, 732827), True, datetime.datetime(2025, 2, 7, 4, 58, 38, 732827), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:58:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:58:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:58:38,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:58:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:58:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:54.423055+00:00 (in 13.998295 seconds)
gcworker stdout | 2025-02-07 04:58:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:10 UTC)" (scheduled at 2025-02-07 04:58:40.424250+00:00)
gcworker stdout | 2025-02-07 04:58:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:58:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904020431, None, 1, 0])
gcworker stdout | 2025-02-07 04:58:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:58:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:58:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:58:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:11.831521+00:00 (in 29.999565 seconds)
autopruneworker stdout | 2025-02-07 04:58:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:11 UTC)" (scheduled at 2025-02-07 04:58:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:58:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900721836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:58:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:58:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:58:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:58:42,332 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:58:42,907 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:58:43,437 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:58:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:58:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:59.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 04:58:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:59 UTC)" (scheduled at 2025-02-07 04:58:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:58:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:58:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:58:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:58:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:58:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:58:44,235 [238] [DEBUG] [app] Starting request: urn:request:df361900-9a7a-4dde-bf7f-2ba366e73e8a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:44,235 [235] [DEBUG] [app] Starting request: urn:request:3ee20d0c-6e40-4600-b8ba-478bb654ebf9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:58:44,238 [245] [DEBUG] [app] Starting request: urn:request:40ea02d5-53d9-4dfb-96a1-7f3510619b60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:44,238 [239] [DEBUG] [app] Starting request: urn:request:b0aee8ec-04ba-43c1-bc2c-bbff3e4b4d2a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:44,239 [239] [DEBUG] [app] Ending request: urn:request:b0aee8ec-04ba-43c1-bc2c-bbff3e4b4d2a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b0aee8ec-04ba-43c1-bc2c-bbff3e4b4d2a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:58:44,239 [245] [DEBUG] [app] Ending request: urn:request:40ea02d5-53d9-4dfb-96a1-7f3510619b60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40ea02d5-53d9-4dfb-96a1-7f3510619b60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 04:58:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:58:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:58:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:58:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:44,242 [238] [DEBUG] [app] Starting request: urn:request:be8dd2e8-e943-402a-b882-20c8d0b01a74 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:44,242 [237] [DEBUG] [app] Starting request: urn:request:5bfead8d-462f-46f3-99e5-1e6f09d75e6e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:44,242 [238] [DEBUG] [app] Ending request: urn:request:be8dd2e8-e943-402a-b882-20c8d0b01a74 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be8dd2e8-e943-402a-b882-20c8d0b01a74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:58:44,243 [237] [DEBUG] [app] Ending request: urn:request:5bfead8d-462f-46f3-99e5-1e6f09d75e6e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5bfead8d-462f-46f3-99e5-1e6f09d75e6e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 04:58:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:58:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:58:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:58:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:44,255 [235] [DEBUG] [app] Ending request: urn:request:3ee20d0c-6e40-4600-b8ba-478bb654ebf9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ee20d0c-6e40-4600-b8ba-478bb654ebf9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:44,255 [238] [DEBUG] [app] Ending request: urn:request:df361900-9a7a-4dde-bf7f-2ba366e73e8a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df361900-9a7a-4dde-bf7f-2ba366e73e8a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:58:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:58:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:58:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:58:44,816 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:58:45,711 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:58:45,751 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:58:46,887 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:58:47,244 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:58:47,255 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:58:47,259 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:58:47,261 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:58:47,264 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:47,349 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:58:47,787 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:58:47,844 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:58:48,018 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:58:48,267 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:58:48,270 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:58:48,273 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:58:48,417 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 04:58:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:58:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:50.814677+00:00 (in 2.082078 seconds)
notificationworker stdout | 2025-02-07 04:58:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:58 UTC)" (scheduled at 2025-02-07 04:58:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:58:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:58:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 48, 732897), True, datetime.datetime(2025, 2, 7, 4, 58, 48, 732897), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:58:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:58:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:58:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:58:58 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:58:48,809 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:58:49,030 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:58:49,220 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:58:49,558 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:58:49,566 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:58:49,572 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:58:49,575 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:58:49,577 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:58:49,580 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:58:49,583 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:58:49,588 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:58:49,590 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:58:50,720 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:58:50,774 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
notificationworker stdout | 2025-02-07 04:58:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:58:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:58:58.732147+00:00 (in 7.916985 seconds)
notificationworker stdout | 2025-02-07 04:58:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:50 UTC)" (scheduled at 2025-02-07 04:58:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:58:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:58:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:50 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 04:58:51,212 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 04:58:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:58:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:10.424250+00:00 (in 16.000686 seconds)
gcworker stdout | 2025-02-07 04:58:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:24 UTC)" (scheduled at 2025-02-07 04:58:54.423055+00:00)
gcworker stdout | 2025-02-07 04:58:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:58:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:24 UTC)" executed successfully
servicekey stdout | 2025-02-07 04:58:55,325 [99] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2025-02-07 04:58:55,326 [99] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:55.325482+00:00 (in 3299.999491 seconds)
servicekey stdout | 2025-02-07 04:58:55,326 [99] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-02-07 05:53:55 UTC)" (scheduled at 2025-02-07 04:58:55.325482+00:00)
servicekey stdout | 2025-02-07 04:58:55,326 [99] [DEBUG] [__main__] Starting automatic refresh of service key rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4 to new expiration 2025-02-07 06:58:55.326291
servicekey stdout | 2025-02-07 04:58:55,326 [99] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4', datetime.datetime(2025, 1, 31, 4, 58, 55, 326481), None, 1, 0])
servicekey stdout | 2025-02-07 04:58:55,333 [99] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise-15131.apps.quaytest-15131.qe.devcluster.openshift.com:443', 'rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4', 'quay', '{"n": "ydZTwWkSP7GLo1VWTY2-tvY2zs-NGpukK9A_Pr1ftW_Qt6JvAWK7vQNC1EA6mULUpUnSEnza3ySALI1yWv9VMgurD8CDnre3MgeFTD3HgFNEnP5af62LG9xYf376yvB9fKjs0XIvd7kbOqdixWiBBR1AGkmOai4M3ehFLQ1vhqcdOlPSPvGgQsRwzTGY78K72BJvpsmB-7yGfu23ebGgLJPjKIwBoWFAsogIVyfxg3kF0x6hFdlu3vEnHpvlHXnAXZyZKfh8rKMbE30HRQGovuegeNsHmy7YpLgtZ-Khw8AKUMVEpkF5u6ljkF6SFN8UuEdD6Kz9s7O1xPBo2iKYqw", "e": "AQAB", "kty": "RSA", "kid": "rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4"}', '{"created_by": "CLI tool"}', datetime.datetime(2025, 2, 7, 3, 15, 32, 537814), datetime.datetime(2025, 2, 7, 6, 58, 55, 326291), None, 2, 2])
servicekey stdout | 2025-02-07 04:58:55,335 [99] [DEBUG] [__main__] Finished automatic refresh of service key rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4 with new expiration 2025-02-07 06:58:55.326291
servicekey stdout | 2025-02-07 04:58:55,335 [99] [DEBUG] [data.database] Disconnecting from database.
servicekey stdout | 2025-02-07 04:58:55,335 [99] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-02-07 05:53:55 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:58:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:58:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:56.118855+00:00 (in 59.999487 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:58:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:56 UTC)" (scheduled at 2025-02-07 04:58:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:58:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:58:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:58:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:37.715878+00:00 (in 41.001900 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:56 UTC)" (scheduled at 2025-02-07 04:58:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:58:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:58:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:58:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:08.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 04:58:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:08 UTC)" (scheduled at 2025-02-07 04:58:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:58:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:58:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 58, 58, 732871), True, datetime.datetime(2025, 2, 7, 4, 58, 58, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:58:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:58:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:58:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:58:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:58:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:14.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 04:58:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:14 UTC)" (scheduled at 2025-02-07 04:58:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:58:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:58:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:58:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:58:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:58:59,235 [238] [DEBUG] [app] Starting request: urn:request:fb3d0211-d897-41ac-83f8-6a0f9cd37de8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:59,235 [235] [DEBUG] [app] Starting request: urn:request:3cd62cb5-bea3-47a6-be0b-d20ad8882f0f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:58:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:58:59,238 [239] [DEBUG] [app] Starting request: urn:request:1c6d0274-5f0b-4f61-8086-73f78a23c3b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:59,238 [245] [DEBUG] [app] Starting request: urn:request:67a53ecd-2e11-4ec5-8455-2ea097b7516f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:58:59,238 [239] [DEBUG] [app] Ending request: urn:request:1c6d0274-5f0b-4f61-8086-73f78a23c3b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c6d0274-5f0b-4f61-8086-73f78a23c3b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:58:59,239 [245] [DEBUG] [app] Ending request: urn:request:67a53ecd-2e11-4ec5-8455-2ea097b7516f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:67a53ecd-2e11-4ec5-8455-2ea097b7516f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:58:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:58:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:58:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:58:59,241 [235] [DEBUG] [app] Starting request: urn:request:0566fba1-ed46-40a1-82c2-0533117ce870 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:59,242 [235] [DEBUG] [app] Ending request: urn:request:0566fba1-ed46-40a1-82c2-0533117ce870 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0566fba1-ed46-40a1-82c2-0533117ce870', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:58:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:58:59,242 [236] [DEBUG] [app] Starting request: urn:request:fb2d1258-7c72-45bf-8595-bf11a90a85dd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:58:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:58:59,242 [236] [DEBUG] [app] Ending request: urn:request:fb2d1258-7c72-45bf-8595-bf11a90a85dd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fb2d1258-7c72-45bf-8595-bf11a90a85dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:58:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:58:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:58:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:58:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:58:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:58:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:58:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:58:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:58:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:58:59,253 [235] [DEBUG] [app] Ending request: urn:request:3cd62cb5-bea3-47a6-be0b-d20ad8882f0f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3cd62cb5-bea3-47a6-be0b-d20ad8882f0f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:58:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:58:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 04:58:59,255 [238] [DEBUG] [app] Ending request: urn:request:fb3d0211-d897-41ac-83f8-6a0f9cd37de8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb3d0211-d897-41ac-83f8-6a0f9cd37de8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:58:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:58:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 04:58:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:58:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:58:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:58:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:04.936688+00:00 (in 4.998270 seconds)
exportactionlogsworker stdout | 2025-02-07 04:58:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:59 UTC)" (scheduled at 2025-02-07 04:58:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:58:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:58:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 04:59:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 04:59:00,193 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 04:59:01,929 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 04:59:03,094 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 04:59:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:59:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:23.420886+00:00 (in 20.001242 seconds)
repositorygcworker stdout | 2025-02-07 04:59:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:03 UTC)" (scheduled at 2025-02-07 04:59:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 04:59:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 04:59:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 3, 419932), True, datetime.datetime(2025, 2, 7, 4, 59, 3, 419932), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 04:59:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 04:59:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 04:59:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 04:59:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:59:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:33.919286+00:00 (in 29.999539 seconds)
buildlogsarchiver stdout | 2025-02-07 04:59:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:33 UTC)" (scheduled at 2025-02-07 04:59:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:59:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 59, 3, 919996), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:59:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:59:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:59:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 04:59:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:59:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:52.937648+00:00 (in 48.000485 seconds)
exportactionlogsworker stdout | 2025-02-07 04:59:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:04 UTC)" (scheduled at 2025-02-07 04:59:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 04:59:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 04:59:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 4, 937526), True, datetime.datetime(2025, 2, 7, 4, 59, 4, 937526), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 04:59:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 04:59:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:59:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:59:07,421 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 04:59:08,465 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:59:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:59:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:18.732147+00:00 (in 9.999547 seconds)
notificationworker stdout | 2025-02-07 04:59:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:18 UTC)" (scheduled at 2025-02-07 04:59:08.732147+00:00)
notificationworker stdout | 2025-02-07 04:59:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:59:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 8, 732928), True, datetime.datetime(2025, 2, 7, 4, 59, 8, 732928), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:59:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:59:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:59:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:59:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:59:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:24.423055+00:00 (in 13.998298 seconds)
gcworker stdout | 2025-02-07 04:59:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:40 UTC)" (scheduled at 2025-02-07 04:59:10.424250+00:00)
gcworker stdout | 2025-02-07 04:59:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:59:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904050431, None, 1, 0])
gcworker stdout | 2025-02-07 04:59:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:59:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:59:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:59:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:41.831521+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2025-02-07 04:59:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:41 UTC)" (scheduled at 2025-02-07 04:59:11.831521+00:00)
autopruneworker stdout | 2025-02-07 04:59:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900751836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:59:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:59:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:59:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:59:12,354 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:59:12,943 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:59:13,473 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:59:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:59:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:29.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 04:59:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:29 UTC)" (scheduled at 2025-02-07 04:59:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:59:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:59:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:59:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:59:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:59:14,235 [236] [DEBUG] [app] Starting request: urn:request:ee91fbf9-2f04-4008-afde-5d1e11c6221b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:14,235 [235] [DEBUG] [app] Starting request: urn:request:75ce9d05-32d8-4c58-b63a-0344f58d8bc3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:14,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:59:14,238 [239] [DEBUG] [app] Starting request: urn:request:0c75a0cc-ce52-4cf0-81fd-99ba29cfa469 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:14,238 [244] [DEBUG] [app] Starting request: urn:request:8547717d-03a0-40f4-8e07-40f78c25536c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:14,238 [239] [DEBUG] [app] Ending request: urn:request:0c75a0cc-ce52-4cf0-81fd-99ba29cfa469 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c75a0cc-ce52-4cf0-81fd-99ba29cfa469', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:59:14,238 [244] [DEBUG] [app] Ending request: urn:request:8547717d-03a0-40f4-8e07-40f78c25536c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8547717d-03a0-40f4-8e07-40f78c25536c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:59:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:59:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:14,241 [235] [DEBUG] [app] Starting request: urn:request:06b6fe9b-27eb-40d5-b44c-9234f54c3466 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:14,241 [237] [DEBUG] [app] Starting request: urn:request:d78503a2-c175-4d0a-a5df-7e643c2a6306 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:14,241 [235] [DEBUG] [app] Ending request: urn:request:06b6fe9b-27eb-40d5-b44c-9234f54c3466 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06b6fe9b-27eb-40d5-b44c-9234f54c3466', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:59:14,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:14,241 [237] [DEBUG] [app] Ending request: urn:request:d78503a2-c175-4d0a-a5df-7e643c2a6306 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d78503a2-c175-4d0a-a5df-7e643c2a6306', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.000)
gunicorn-web stdout | 2025-02-07 04:59:14,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 04:59:14,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:14,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:14,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:14,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:14,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:14,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:14,252 [235] [DEBUG] [app] Ending request: urn:request:75ce9d05-32d8-4c58-b63a-0344f58d8bc3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75ce9d05-32d8-4c58-b63a-0344f58d8bc3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:59:14,253 [236] [DEBUG] [app] Ending request: urn:request:ee91fbf9-2f04-4008-afde-5d1e11c6221b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee91fbf9-2f04-4008-afde-5d1e11c6221b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:14,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:59:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 04:59:14,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 04:59:14,843 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:59:15,747 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:59:15,767 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:59:16,923 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:59:17,252 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:59:17,261 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:59:17,264 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:59:17,268 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:59:17,273 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:17,362 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:59:17,823 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:59:17,853 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:59:18,026 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:59:18,275 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:59:18,279 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:59:18,282 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:59:18,447 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 04:59:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:59:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:28.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 04:59:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:28 UTC)" (scheduled at 2025-02-07 04:59:18.732147+00:00)
notificationworker stdout | 2025-02-07 04:59:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:59:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 18, 732851), True, datetime.datetime(2025, 2, 7, 4, 59, 18, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:59:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:59:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:59:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:59:18,845 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:59:19,056 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:59:19,256 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:59:19,567 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:59:19,574 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:59:19,581 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:59:19,583 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:59:19,586 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:59:19,589 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:59:19,593 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:59:19,596 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:59:19,598 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:59:20,756 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 04:59:20,811 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:59:21,248 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 04:59:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 04:59:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:03.419110+00:00 (in 39.997745 seconds)
repositorygcworker stdout | 2025-02-07 04:59:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:23 UTC)" (scheduled at 2025-02-07 04:59:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 04:59:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 04:59:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:59:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:59:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:40.424250+00:00 (in 16.000737 seconds)
gcworker stdout | 2025-02-07 04:59:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:54 UTC)" (scheduled at 2025-02-07 04:59:24.423055+00:00)
gcworker stdout | 2025-02-07 04:59:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:59:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 04:59:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:59:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:59:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:38.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 04:59:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:38 UTC)" (scheduled at 2025-02-07 04:59:28.732147+00:00)
notificationworker stdout | 2025-02-07 04:59:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:59:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 28, 732869), True, datetime.datetime(2025, 2, 7, 4, 59, 28, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:59:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:59:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:59:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:59:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:59:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:44.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 04:59:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:44 UTC)" (scheduled at 2025-02-07 04:59:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:59:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:59:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:59:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:59:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:59:29,235 [238] [DEBUG] [app] Starting request: urn:request:eac7b314-a587-4553-857d-8980d6278072 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:29,235 [235] [DEBUG] [app] Starting request: urn:request:69f77f7d-9cc4-46dd-89a8-3e5a913508c5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:59:29,239 [246] [DEBUG] [app] Starting request: urn:request:2db6bc46-ce90-497c-b3a7-098f46a6c66f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:29,239 [239] [DEBUG] [app] Starting request: urn:request:a5b1c82b-2e55-4191-a981-9165c3401ab8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:29,239 [246] [DEBUG] [app] Ending request: urn:request:2db6bc46-ce90-497c-b3a7-098f46a6c66f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2db6bc46-ce90-497c-b3a7-098f46a6c66f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:59:29,239 [239] [DEBUG] [app] Ending request: urn:request:a5b1c82b-2e55-4191-a981-9165c3401ab8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5b1c82b-2e55-4191-a981-9165c3401ab8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:59:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:59:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 04:59:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:29,242 [238] [DEBUG] [app] Starting request: urn:request:a7ae02d3-75aa-4b37-884f-a3df9b567d50 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:29,242 [237] [DEBUG] [app] Starting request: urn:request:a89a3eef-f278-434f-a40f-5a88eae17714 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:29,242 [238] [DEBUG] [app] Ending request: urn:request:a7ae02d3-75aa-4b37-884f-a3df9b567d50 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7ae02d3-75aa-4b37-884f-a3df9b567d50', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:59:29,243 [237] [DEBUG] [app] Ending request: urn:request:a89a3eef-f278-434f-a40f-5a88eae17714 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a89a3eef-f278-434f-a40f-5a88eae17714', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:59:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:59:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:59:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:29,254 [238] [DEBUG] [app] Ending request: urn:request:eac7b314-a587-4553-857d-8980d6278072 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eac7b314-a587-4553-857d-8980d6278072', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:59:29,254 [235] [DEBUG] [app] Ending request: urn:request:69f77f7d-9cc4-46dd-89a8-3e5a913508c5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69f77f7d-9cc4-46dd-89a8-3e5a913508c5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:59:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 04:59:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
storagereplication stdout | 2025-02-07 04:59:30,209 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 04:59:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:59:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:32.625945+00:00 (in 1.004279 seconds)
namespacegcworker stdout | 2025-02-07 04:59:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:31 UTC)" (scheduled at 2025-02-07 04:59:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 04:59:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 04:59:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 31, 621953), True, datetime.datetime(2025, 2, 7, 4, 59, 31, 621953), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 04:59:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 04:59:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 04:59:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 04:59:31,962 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 04:59:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 04:59:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:31.621131+00:00 (in 58.994730 seconds)
namespacegcworker stdout | 2025-02-07 04:59:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:32 UTC)" (scheduled at 2025-02-07 04:59:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 04:59:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 04:59:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 04:59:33,117 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 04:59:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 04:59:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:03.919286+00:00 (in 29.999546 seconds)
buildlogsarchiver stdout | 2025-02-07 04:59:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:03 UTC)" (scheduled at 2025-02-07 04:59:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 04:59:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 4, 59, 33, 920003), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 04:59:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 04:59:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 04:59:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 04:59:37,435 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:56.713547+00:00 (in 18.997194 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:37 UTC)" (scheduled at 2025-02-07 04:59:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:59:38,501 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 04:59:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:59:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:48.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 04:59:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:48 UTC)" (scheduled at 2025-02-07 04:59:38.732147+00:00)
notificationworker stdout | 2025-02-07 04:59:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:59:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 38, 732894), True, datetime.datetime(2025, 2, 7, 4, 59, 38, 732894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:59:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:59:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:59:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:59:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:59:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:54.423055+00:00 (in 13.998323 seconds)
gcworker stdout | 2025-02-07 04:59:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:10 UTC)" (scheduled at 2025-02-07 04:59:40.424250+00:00)
gcworker stdout | 2025-02-07 04:59:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 04:59:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904080431, None, 1, 0])
gcworker stdout | 2025-02-07 04:59:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 04:59:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 04:59:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 04:59:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:11.831521+00:00 (in 29.999574 seconds)
autopruneworker stdout | 2025-02-07 04:59:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:11 UTC)" (scheduled at 2025-02-07 04:59:41.831521+00:00)
autopruneworker stdout | 2025-02-07 04:59:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900781836, None, 1, 0])
autopruneworker stdout | 2025-02-07 04:59:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 04:59:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 04:59:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:59:42,387 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 04:59:42,975 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 04:59:43,502 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 04:59:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:59:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:59.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 04:59:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:59 UTC)" (scheduled at 2025-02-07 04:59:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:59:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:59:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:59:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:59:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 04:59:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:59:44,236 [238] [DEBUG] [app] Starting request: urn:request:84dff486-0d88-4815-b94a-1477cc757241 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:44,236 [235] [DEBUG] [app] Starting request: urn:request:cd7587da-88ee-4ec6-9c7c-6c2c2d4687b3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:59:44,239 [244] [DEBUG] [app] Starting request: urn:request:111e6b7c-d097-4ca0-81f7-b0c7838adc8a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:44,239 [239] [DEBUG] [app] Starting request: urn:request:efb93fd3-88f6-4e43-8906-f306e9255b90 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:44,240 [239] [DEBUG] [app] Ending request: urn:request:efb93fd3-88f6-4e43-8906-f306e9255b90 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:efb93fd3-88f6-4e43-8906-f306e9255b90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:59:44,240 [244] [DEBUG] [app] Ending request: urn:request:111e6b7c-d097-4ca0-81f7-b0c7838adc8a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:111e6b7c-d097-4ca0-81f7-b0c7838adc8a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 04:59:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:59:44,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:44,243 [238] [DEBUG] [app] Starting request: urn:request:3d839dd9-f30a-4690-8319-7662c762141e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:44,243 [237] [DEBUG] [app] Starting request: urn:request:d124b0bd-0ea5-46ca-974c-f5dc78132aa7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:44,243 [238] [DEBUG] [app] Ending request: urn:request:3d839dd9-f30a-4690-8319-7662c762141e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d839dd9-f30a-4690-8319-7662c762141e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:59:44,243 [237] [DEBUG] [app] Ending request: urn:request:d124b0bd-0ea5-46ca-974c-f5dc78132aa7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d124b0bd-0ea5-46ca-974c-f5dc78132aa7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:59:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:59:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:44,254 [235] [DEBUG] [app] Ending request: urn:request:cd7587da-88ee-4ec6-9c7c-6c2c2d4687b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd7587da-88ee-4ec6-9c7c-6c2c2d4687b3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:44,254 [238] [DEBUG] [app] Ending request: urn:request:84dff486-0d88-4815-b94a-1477cc757241 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:84dff486-0d88-4815-b94a-1477cc757241', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:59:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:59:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 04:59:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 04:59:44,853 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 04:59:45,763 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 04:59:45,803 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 04:59:46,959 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 04:59:47,260 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 04:59:47,268 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 04:59:47,272 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 04:59:47,274 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 04:59:47,279 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:47,398 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 04:59:47,844 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 04:59:47,888 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 04:59:48,049 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 04:59:48,283 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 04:59:48,286 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 04:59:48,289 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 04:59:48,482 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 04:59:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:59:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:50.814677+00:00 (in 2.082096 seconds)
notificationworker stdout | 2025-02-07 04:59:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:58 UTC)" (scheduled at 2025-02-07 04:59:48.732147+00:00)
notificationworker stdout | 2025-02-07 04:59:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:59:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 48, 732858), True, datetime.datetime(2025, 2, 7, 4, 59, 48, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:59:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:59:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:59:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 04:59:58 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:59:48,881 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 04:59:49,065 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 04:59:49,292 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 04:59:49,574 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 04:59:49,582 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 04:59:49,587 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 04:59:49,590 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 04:59:49,594 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 04:59:49,597 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 04:59:49,601 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 04:59:49,604 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 04:59:49,607 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 04:59:50,778 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
notificationworker stdout | 2025-02-07 04:59:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:59:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:58.732147+00:00 (in 7.917010 seconds)
notificationworker stdout | 2025-02-07 04:59:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:50 UTC)" (scheduled at 2025-02-07 04:59:50.814677+00:00)
notificationworker stdout | 2025-02-07 04:59:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 04:59:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:50 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 04:59:50,843 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 04:59:51,285 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
exportactionlogsworker stdout | 2025-02-07 04:59:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:59:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 04:59:59.937899+00:00 (in 6.999753 seconds)
exportactionlogsworker stdout | 2025-02-07 04:59:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:04:52 UTC)" (scheduled at 2025-02-07 04:59:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 04:59:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 4, 59, 52, 938448), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:59:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 59, 52, 938448), True, datetime.datetime(2025, 2, 7, 4, 59, 52, 938448), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:59:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 4, 59, 52, 938448), True, datetime.datetime(2025, 2, 7, 4, 59, 52, 938448), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 4, 59, 52, 938448), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 04:59:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 04:59:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:04:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 04:59:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 04:59:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:10.424250+00:00 (in 16.000732 seconds)
gcworker stdout | 2025-02-07 04:59:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:24 UTC)" (scheduled at 2025-02-07 04:59:54.423055+00:00)
gcworker stdout | 2025-02-07 04:59:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 04:59:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 04:59:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 04:59:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:56.118855+00:00 (in 59.999453 seconds)
quotaregistrysizeworker stdout | 2025-02-07 04:59:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:56 UTC)" (scheduled at 2025-02-07 04:59:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 04:59:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 04:59:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 04:59:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:37.715878+00:00 (in 41.001853 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:56 UTC)" (scheduled at 2025-02-07 04:59:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 04:59:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 04:59:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 04:59:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:08.732147+00:00 (in 9.999481 seconds)
notificationworker stdout | 2025-02-07 04:59:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:08 UTC)" (scheduled at 2025-02-07 04:59:58.732147+00:00)
notificationworker stdout | 2025-02-07 04:59:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 04:59:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 4, 59, 58, 732978), True, datetime.datetime(2025, 2, 7, 4, 59, 58, 732978), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 04:59:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 04:59:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 04:59:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 04:59:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 04:59:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:14.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 04:59:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:14 UTC)" (scheduled at 2025-02-07 04:59:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 04:59:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 04:59:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 04:59:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 04:59:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 04:59:59,235 [238] [DEBUG] [app] Starting request: urn:request:66bb5a37-79d3-47a8-8e62-cffec4a2aa4e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:59,235 [237] [DEBUG] [app] Starting request: urn:request:25e7a57d-f66d-461e-8d19-fb833e2be0af (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 04:59:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 04:59:59,238 [239] [DEBUG] [app] Starting request: urn:request:47998ce6-c0ea-415e-96c2-54505df8204a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:59,238 [246] [DEBUG] [app] Starting request: urn:request:0cbf93a4-a280-431a-ab25-61e771f00bd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 04:59:59,239 [239] [DEBUG] [app] Ending request: urn:request:47998ce6-c0ea-415e-96c2-54505df8204a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:47998ce6-c0ea-415e-96c2-54505df8204a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:59:59,239 [246] [DEBUG] [app] Ending request: urn:request:0cbf93a4-a280-431a-ab25-61e771f00bd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0cbf93a4-a280-431a-ab25-61e771f00bd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 04:59:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 04:59:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 04:59:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 04:59:59,241 [237] [DEBUG] [app] Starting request: urn:request:6d8374cf-bd73-4007-bd85-048cf86dab91 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:59,242 [237] [DEBUG] [app] Ending request: urn:request:6d8374cf-bd73-4007-bd85-048cf86dab91 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6d8374cf-bd73-4007-bd85-048cf86dab91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:59:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 04:59:59,242 [236] [DEBUG] [app] Starting request: urn:request:cfd90747-a784-43ab-b9c1-70983651b5cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 04:59:59,243 [236] [DEBUG] [app] Ending request: urn:request:cfd90747-a784-43ab-b9c1-70983651b5cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cfd90747-a784-43ab-b9c1-70983651b5cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 04:59:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:04:59:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 04:59:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:04:59:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 04:59:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 04:59:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 04:59:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 04:59:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 04:59:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 04:59:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 04:59:59,253 [237] [DEBUG] [app] Ending request: urn:request:25e7a57d-f66d-461e-8d19-fb833e2be0af (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:25e7a57d-f66d-461e-8d19-fb833e2be0af', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 04:59:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
gunicorn-web stdout | 2025-02-07 04:59:59,254 [238] [DEBUG] [app] Ending request: urn:request:66bb5a37-79d3-47a8-8e62-cffec4a2aa4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66bb5a37-79d3-47a8-8e62-cffec4a2aa4e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 04:59:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:04:59:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 04:59:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:04:59:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 04:59:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 04:59:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:04.936688+00:00 (in 4.998267 seconds)
exportactionlogsworker stdout | 2025-02-07 04:59:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:59 UTC)" (scheduled at 2025-02-07 04:59:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 04:59:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 04:59:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:00:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:00:00,242 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:00:01,997 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:00:03,148 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 05:00:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:00:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:23.420886+00:00 (in 20.001306 seconds)
repositorygcworker stdout | 2025-02-07 05:00:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:03 UTC)" (scheduled at 2025-02-07 05:00:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:00:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:00:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 3, 419862), True, datetime.datetime(2025, 2, 7, 5, 0, 3, 419862), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:00:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:00:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:00:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:00:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:00:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:33.919286+00:00 (in 29.999537 seconds)
buildlogsarchiver stdout | 2025-02-07 05:00:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:33 UTC)" (scheduled at 2025-02-07 05:00:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:00:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 0, 3, 920035), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:00:03,928 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:00:03,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:00:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:00:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:00:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:59.937899+00:00 (in 55.000783 seconds)
exportactionlogsworker stdout | 2025-02-07 05:00:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:04 UTC)" (scheduled at 2025-02-07 05:00:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:00:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:00:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 4, 937446), True, datetime.datetime(2025, 2, 7, 5, 0, 4, 937446), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:00:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:00:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:00:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:00:07,445 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:00:08,511 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:00:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:00:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:18.732147+00:00 (in 9.999504 seconds)
notificationworker stdout | 2025-02-07 05:00:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:18 UTC)" (scheduled at 2025-02-07 05:00:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:00:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:00:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 8, 732926), True, datetime.datetime(2025, 2, 7, 5, 0, 8, 732926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:00:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:00:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:00:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:00:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:00:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:24.423055+00:00 (in 13.998337 seconds)
gcworker stdout | 2025-02-07 05:00:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:40 UTC)" (scheduled at 2025-02-07 05:00:10.424250+00:00)
gcworker stdout | 2025-02-07 05:00:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:00:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904110431, None, 1, 0])
gcworker stdout | 2025-02-07 05:00:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:00:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:00:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:00:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:41.831521+00:00 (in 29.999572 seconds)
autopruneworker stdout | 2025-02-07 05:00:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:41 UTC)" (scheduled at 2025-02-07 05:00:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:00:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900811836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:00:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:00:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:00:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:00:12,423 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:00:13,002 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:00:13,539 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:00:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:00:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:29.225378+00:00 (in 14.999538 seconds)
quotatotalworker stdout | 2025-02-07 05:00:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:29 UTC)" (scheduled at 2025-02-07 05:00:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:00:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:00:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:00:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:00:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:00:14,235 [237] [DEBUG] [app] Starting request: urn:request:b9e1e16e-fd15-49dc-999e-4442966b5147 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:14,235 [235] [DEBUG] [app] Starting request: urn:request:e43214eb-dcf3-440b-9518-1381ba7645f6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:00:14,238 [239] [DEBUG] [app] Starting request: urn:request:2f3f4a25-bdf1-44ab-9612-827e5287864b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:14,238 [244] [DEBUG] [app] Starting request: urn:request:0326e13f-5653-4462-a295-d92a6adfa5b7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:14,239 [239] [DEBUG] [app] Ending request: urn:request:2f3f4a25-bdf1-44ab-9612-827e5287864b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f3f4a25-bdf1-44ab-9612-827e5287864b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:00:14,239 [244] [DEBUG] [app] Ending request: urn:request:0326e13f-5653-4462-a295-d92a6adfa5b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0326e13f-5653-4462-a295-d92a6adfa5b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:00:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:00:14,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:00:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:00:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:14,242 [237] [DEBUG] [app] Starting request: urn:request:f1a38806-169d-4505-ac54-0644d8888ab6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:00:14,242 [238] [DEBUG] [app] Starting request: urn:request:ba58fb09-de7d-49d7-bbb9-73996af0519e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:00:14,242 [237] [DEBUG] [app] Ending request: urn:request:f1a38806-169d-4505-ac54-0644d8888ab6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f1a38806-169d-4505-ac54-0644d8888ab6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:00:14,242 [238] [DEBUG] [app] Ending request: urn:request:ba58fb09-de7d-49d7-bbb9-73996af0519e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba58fb09-de7d-49d7-bbb9-73996af0519e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:00:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:00:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:00:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:14,254 [237] [DEBUG] [app] Ending request: urn:request:b9e1e16e-fd15-49dc-999e-4442966b5147 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b9e1e16e-fd15-49dc-999e-4442966b5147', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:14,254 [235] [DEBUG] [app] Ending request: urn:request:e43214eb-dcf3-440b-9518-1381ba7645f6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e43214eb-dcf3-440b-9518-1381ba7645f6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:00:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
expiredappspecifictokenworker stdout | 2025-02-07 05:00:14,889 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:00:15,800 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:00:15,839 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:00:16,996 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:00:17,268 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:00:17,276 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:00:17,279 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:00:17,282 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:00:17,285 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:17,418 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:00:17,880 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:00:17,917 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:00:18,085 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:00:18,293 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:00:18,297 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:00:18,300 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:00:18,518 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:00:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:00:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:28.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 05:00:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:28 UTC)" (scheduled at 2025-02-07 05:00:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:00:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:00:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 18, 732894), True, datetime.datetime(2025, 2, 7, 5, 0, 18, 732894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:00:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:00:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:00:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:00:18,917 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:00:19,099 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:00:19,328 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:00:19,584 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:00:19,590 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:00:19,593 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:00:19,596 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:00:19,600 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:00:19,607 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:00:19,610 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:00:19,615 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:00:19,618 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 05:00:20,814 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:00:20,852 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:00:21,314 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:00:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:00:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:03.419110+00:00 (in 39.997752 seconds)
repositorygcworker stdout | 2025-02-07 05:00:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:23 UTC)" (scheduled at 2025-02-07 05:00:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:00:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:00:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:00:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:00:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:40.424250+00:00 (in 16.000702 seconds)
gcworker stdout | 2025-02-07 05:00:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:54 UTC)" (scheduled at 2025-02-07 05:00:24.423055+00:00)
gcworker stdout | 2025-02-07 05:00:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:00:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:00:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:00:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:00:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:38.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 05:00:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:38 UTC)" (scheduled at 2025-02-07 05:00:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:00:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:00:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 28, 732878), True, datetime.datetime(2025, 2, 7, 5, 0, 28, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:00:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:00:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:00:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:00:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:00:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:44.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 05:00:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:44 UTC)" (scheduled at 2025-02-07 05:00:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:00:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:00:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:00:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:00:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:00:29,235 [237] [DEBUG] [app] Starting request: urn:request:c4c8750a-065e-432f-a9c5-492da758fb6e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:29,235 [235] [DEBUG] [app] Starting request: urn:request:a6c3e6a9-27db-46ca-b8ac-31161cd47c6b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:00:29,238 [246] [DEBUG] [app] Starting request: urn:request:7f02f00a-79a3-4b8a-a183-6bb87f46a58a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:29,238 [245] [DEBUG] [app] Starting request: urn:request:401e4e4c-0c93-4493-a62e-749fa736cfa0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:29,239 [246] [DEBUG] [app] Ending request: urn:request:7f02f00a-79a3-4b8a-a183-6bb87f46a58a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f02f00a-79a3-4b8a-a183-6bb87f46a58a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:00:29,239 [245] [DEBUG] [app] Ending request: urn:request:401e4e4c-0c93-4493-a62e-749fa736cfa0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:401e4e4c-0c93-4493-a62e-749fa736cfa0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:00:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:00:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:00:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:00:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:29,242 [235] [DEBUG] [app] Starting request: urn:request:6354cd1a-7298-4a31-be82-adc75ba6e228 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:00:29,242 [235] [DEBUG] [app] Ending request: urn:request:6354cd1a-7298-4a31-be82-adc75ba6e228 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6354cd1a-7298-4a31-be82-adc75ba6e228', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:00:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:00:29,242 [236] [DEBUG] [app] Starting request: urn:request:73e6d4a2-1497-4bba-ae6c-63acd9b15ef9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:00:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:29,243 [236] [DEBUG] [app] Ending request: urn:request:73e6d4a2-1497-4bba-ae6c-63acd9b15ef9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73e6d4a2-1497-4bba-ae6c-63acd9b15ef9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:00:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:00:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:00:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:29,254 [235] [DEBUG] [app] Ending request: urn:request:a6c3e6a9-27db-46ca-b8ac-31161cd47c6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6c3e6a9-27db-46ca-b8ac-31161cd47c6b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:29,254 [237] [DEBUG] [app] Ending request: urn:request:c4c8750a-065e-432f-a9c5-492da758fb6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4c8750a-065e-432f-a9c5-492da758fb6e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:00:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
storagereplication stdout | 2025-02-07 05:00:30,278 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:00:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:00:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:32.625945+00:00 (in 1.004395 seconds)
namespacegcworker stdout | 2025-02-07 05:00:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:31 UTC)" (scheduled at 2025-02-07 05:00:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:00:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:00:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 31, 621820), True, datetime.datetime(2025, 2, 7, 5, 0, 31, 621820), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:00:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:00:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:00:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:00:32,029 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:00:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:00:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:31.621131+00:00 (in 58.994749 seconds)
namespacegcworker stdout | 2025-02-07 05:00:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:32 UTC)" (scheduled at 2025-02-07 05:00:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:00:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:00:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:00:33,184 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:00:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:00:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:03.919286+00:00 (in 29.999516 seconds)
buildlogsarchiver stdout | 2025-02-07 05:00:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:03 UTC)" (scheduled at 2025-02-07 05:00:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:00:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 0, 33, 920053), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:00:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:00:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:00:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:00:37,481 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:56.713547+00:00 (in 18.997150 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:37 UTC)" (scheduled at 2025-02-07 05:00:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:00:38,548 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:00:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:00:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:48.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 05:00:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:48 UTC)" (scheduled at 2025-02-07 05:00:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:00:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:00:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 38, 732920), True, datetime.datetime(2025, 2, 7, 5, 0, 38, 732920), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:00:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:00:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:00:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:00:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:00:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:54.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 05:00:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:10 UTC)" (scheduled at 2025-02-07 05:00:40.424250+00:00)
gcworker stdout | 2025-02-07 05:00:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:00:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904140431, None, 1, 0])
gcworker stdout | 2025-02-07 05:00:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:00:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:00:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:00:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:11.831521+00:00 (in 29.999563 seconds)
autopruneworker stdout | 2025-02-07 05:00:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:11 UTC)" (scheduled at 2025-02-07 05:00:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:00:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900841836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:00:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:00:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:00:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:00:42,432 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:00:43,018 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:00:43,575 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:00:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:00:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:59.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 05:00:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:59 UTC)" (scheduled at 2025-02-07 05:00:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:00:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:00:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:00:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:00:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:00:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:00:44,236 [237] [DEBUG] [app] Starting request: urn:request:2e2a451b-a37c-4789-9fcf-d46924a46f53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:44,236 [235] [DEBUG] [app] Starting request: urn:request:6cdb2d52-5eb5-45c0-80da-db783eb8e2bb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:44,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:00:44,240 [245] [DEBUG] [app] Starting request: urn:request:87af9f09-beb2-45c3-9219-6af0f93681c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:44,240 [246] [DEBUG] [app] Starting request: urn:request:e4bdd2a2-71fb-47a4-9b7e-d64ef924a2b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:44,240 [245] [DEBUG] [app] Ending request: urn:request:87af9f09-beb2-45c3-9219-6af0f93681c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:87af9f09-beb2-45c3-9219-6af0f93681c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:00:44,240 [246] [DEBUG] [app] Ending request: urn:request:e4bdd2a2-71fb-47a4-9b7e-d64ef924a2b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e4bdd2a2-71fb-47a4-9b7e-d64ef924a2b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:00:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:00:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:00:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:44,243 [238] [DEBUG] [app] Starting request: urn:request:91e07824-9bb7-4848-bec5-f5ec290471e9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:00:44,243 [238] [DEBUG] [app] Ending request: urn:request:91e07824-9bb7-4848-bec5-f5ec290471e9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91e07824-9bb7-4848-bec5-f5ec290471e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:00:44,243 [235] [DEBUG] [app] Starting request: urn:request:9426bc86-6868-4443-b0c0-6e9a874ce2c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:00:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:00:44,244 [235] [DEBUG] [app] Ending request: urn:request:9426bc86-6868-4443-b0c0-6e9a874ce2c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9426bc86-6868-4443-b0c0-6e9a874ce2c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:00:44,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:00:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:44,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:44,255 [235] [DEBUG] [app] Ending request: urn:request:6cdb2d52-5eb5-45c0-80da-db783eb8e2bb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cdb2d52-5eb5-45c0-80da-db783eb8e2bb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:44,255 [237] [DEBUG] [app] Ending request: urn:request:2e2a451b-a37c-4789-9fcf-d46924a46f53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e2a451b-a37c-4789-9fcf-d46924a46f53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:00:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
expiredappspecifictokenworker stdout | 2025-02-07 05:00:44,919 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:00:45,832 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:00:45,875 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:00:47,032 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:00:47,276 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:00:47,286 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:00:47,289 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:00:47,292 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:00:47,296 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:47,432 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:00:47,907 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:00:47,924 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:00:48,121 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:00:48,301 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:00:48,305 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:00:48,307 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:00:48,555 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:00:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:00:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:50.814677+00:00 (in 2.082062 seconds)
notificationworker stdout | 2025-02-07 05:00:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:58 UTC)" (scheduled at 2025-02-07 05:00:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:00:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:00:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 48, 732924), True, datetime.datetime(2025, 2, 7, 5, 0, 48, 732924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:00:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:00:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:00:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:00:58 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:00:48,931 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:00:49,130 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:00:49,341 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:00:49,593 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:00:49,597 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:00:49,599 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:00:49,606 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:00:49,610 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:00:49,617 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:00:49,622 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:00:49,624 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:00:49,628 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
notificationworker stdout | 2025-02-07 05:00:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:00:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:00:58.732147+00:00 (in 7.916964 seconds)
notificationworker stdout | 2025-02-07 05:00:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:50 UTC)" (scheduled at 2025-02-07 05:00:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:00:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:00:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:00:50,831 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:00:50,887 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:00:51,350 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:00:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:00:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 05:00:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:24 UTC)" (scheduled at 2025-02-07 05:00:54.423055+00:00)
gcworker stdout | 2025-02-07 05:00:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:00:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:00:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:00:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:56.118855+00:00 (in 59.999462 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:00:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:56 UTC)" (scheduled at 2025-02-07 05:00:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:00:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:00:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:00:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:37.715878+00:00 (in 41.001866 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:56 UTC)" (scheduled at 2025-02-07 05:00:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:00:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:00:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:00:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:08.732147+00:00 (in 9.999510 seconds)
notificationworker stdout | 2025-02-07 05:00:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:08 UTC)" (scheduled at 2025-02-07 05:00:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:00:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:00:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 0, 58, 732943), True, datetime.datetime(2025, 2, 7, 5, 0, 58, 732943), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:00:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:00:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:00:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:00:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:00:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:14.225378+00:00 (in 14.999556 seconds)
quotatotalworker stdout | 2025-02-07 05:00:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:14 UTC)" (scheduled at 2025-02-07 05:00:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:00:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:00:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:00:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:00:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:00:59,236 [237] [DEBUG] [app] Starting request: urn:request:11444bf4-12e4-4acf-ac9c-9ec279d12cff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:59,236 [238] [DEBUG] [app] Starting request: urn:request:a7fdaae3-e69d-4180-9bf3-195b8fd2ee04 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:00:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:00:59,238 [244] [DEBUG] [app] Starting request: urn:request:1502d477-4a10-4bb9-b452-1133904fac5f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:59,239 [239] [DEBUG] [app] Starting request: urn:request:69a3e941-f8e1-4250-907b-55b841e06582 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:00:59,239 [244] [DEBUG] [app] Ending request: urn:request:1502d477-4a10-4bb9-b452-1133904fac5f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1502d477-4a10-4bb9-b452-1133904fac5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:00:59,239 [239] [DEBUG] [app] Ending request: urn:request:69a3e941-f8e1-4250-907b-55b841e06582 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:69a3e941-f8e1-4250-907b-55b841e06582', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:00:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:00:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:00:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:00:59,242 [235] [DEBUG] [app] Starting request: urn:request:599747fa-2684-4e1b-9a16-629f0375c70b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:00:59,242 [235] [DEBUG] [app] Ending request: urn:request:599747fa-2684-4e1b-9a16-629f0375c70b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:599747fa-2684-4e1b-9a16-629f0375c70b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:00:59,242 [238] [DEBUG] [app] Starting request: urn:request:2c479e92-a732-447c-b526-0712566afa9c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:00:59,242 [238] [DEBUG] [app] Ending request: urn:request:2c479e92-a732-447c-b526-0712566afa9c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2c479e92-a732-447c-b526-0712566afa9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:00:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:00:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:00:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:00:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:00:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:00:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:00:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:00:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:00:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:00:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:00:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:00:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:00:59,254 [238] [DEBUG] [app] Ending request: urn:request:a7fdaae3-e69d-4180-9bf3-195b8fd2ee04 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7fdaae3-e69d-4180-9bf3-195b8fd2ee04', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:59,254 [237] [DEBUG] [app] Ending request: urn:request:11444bf4-12e4-4acf-ac9c-9ec279d12cff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11444bf4-12e4-4acf-ac9c-9ec279d12cff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:00:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:00:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:00:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:00:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:00:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:00:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:00:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:04.936688+00:00 (in 4.998320 seconds)
exportactionlogsworker stdout | 2025-02-07 05:00:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:59 UTC)" (scheduled at 2025-02-07 05:00:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:00:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:00:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:01:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:01:00,292 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:01:02,061 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:01:03,218 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 05:01:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:01:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:23.420886+00:00 (in 20.001303 seconds)
repositorygcworker stdout | 2025-02-07 05:01:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:03 UTC)" (scheduled at 2025-02-07 05:01:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:01:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:01:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 3, 419862), True, datetime.datetime(2025, 2, 7, 5, 1, 3, 419862), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:01:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:01:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:01:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:01:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:01:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:33.919286+00:00 (in 29.999493 seconds)
buildlogsarchiver stdout | 2025-02-07 05:01:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:33 UTC)" (scheduled at 2025-02-07 05:01:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:01:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 1, 3, 920042), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:01:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:01:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:01:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:01:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:01:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:59.937899+00:00 (in 55.000718 seconds)
exportactionlogsworker stdout | 2025-02-07 05:01:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:04 UTC)" (scheduled at 2025-02-07 05:01:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:01:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:01:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 4, 937498), True, datetime.datetime(2025, 2, 7, 5, 1, 4, 937498), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:01:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:01:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:01:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:01:07,517 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:01:08,563 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:01:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:01:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:18.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 05:01:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:18 UTC)" (scheduled at 2025-02-07 05:01:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:01:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:01:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 8, 732844), True, datetime.datetime(2025, 2, 7, 5, 1, 8, 732844), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:01:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:01:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:01:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:01:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:01:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:24.423055+00:00 (in 13.998353 seconds)
gcworker stdout | 2025-02-07 05:01:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:40 UTC)" (scheduled at 2025-02-07 05:01:10.424250+00:00)
gcworker stdout | 2025-02-07 05:01:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:01:10,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904170431, None, 1, 0])
gcworker stdout | 2025-02-07 05:01:10,434 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:01:10,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:01:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:01:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:41.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 05:01:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:41 UTC)" (scheduled at 2025-02-07 05:01:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:01:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900871836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:01:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:01:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:01:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:01:12,468 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:01:13,037 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:01:13,611 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:01:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:01:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:29.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 05:01:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:29 UTC)" (scheduled at 2025-02-07 05:01:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:01:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:01:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:01:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:01:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:01:14,236 [237] [DEBUG] [app] Starting request: urn:request:f7a835d1-f64c-4a2d-8a68-37b3f9528a5d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:14,236 [235] [DEBUG] [app] Starting request: urn:request:75ca0c99-4108-4c63-859b-684276e0461c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:01:14,239 [246] [DEBUG] [app] Starting request: urn:request:e1f654a4-6375-48f2-a478-bea144713e71 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:14,239 [239] [DEBUG] [app] Starting request: urn:request:1161254c-65b1-4905-803a-0c8f3ab7bb5e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:14,239 [246] [DEBUG] [app] Ending request: urn:request:e1f654a4-6375-48f2-a478-bea144713e71 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e1f654a4-6375-48f2-a478-bea144713e71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:01:14,239 [239] [DEBUG] [app] Ending request: urn:request:1161254c-65b1-4905-803a-0c8f3ab7bb5e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1161254c-65b1-4905-803a-0c8f3ab7bb5e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:01:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:01:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:14,243 [237] [DEBUG] [app] Starting request: urn:request:694c0fbe-9f5c-42ae-a493-cdda91d9c809 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:14,243 [237] [DEBUG] [app] Ending request: urn:request:694c0fbe-9f5c-42ae-a493-cdda91d9c809 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:694c0fbe-9f5c-42ae-a493-cdda91d9c809', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:01:14,244 [238] [DEBUG] [app] Starting request: urn:request:845907f4-6819-4887-9ab1-a20ff8735439 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:01:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:14,244 [238] [DEBUG] [app] Ending request: urn:request:845907f4-6819-4887-9ab1-a20ff8735439 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:845907f4-6819-4887-9ab1-a20ff8735439', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:01:14,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:14,245 [237] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:01:14,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:14,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:01:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:14,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:14,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:14,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:14,255 [237] [DEBUG] [app] Ending request: urn:request:f7a835d1-f64c-4a2d-8a68-37b3f9528a5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f7a835d1-f64c-4a2d-8a68-37b3f9528a5d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:01:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:01:14,257 [235] [DEBUG] [app] Ending request: urn:request:75ca0c99-4108-4c63-859b-684276e0461c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:75ca0c99-4108-4c63-859b-684276e0461c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:14,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:01:14,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
expiredappspecifictokenworker stdout | 2025-02-07 05:01:14,954 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:01:15,866 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:01:15,887 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:01:17,069 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:01:17,285 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:01:17,294 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:01:17,297 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:01:17,301 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:01:17,304 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:17,468 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:01:17,917 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:01:17,960 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:01:18,158 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:01:18,309 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:01:18,311 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:01:18,314 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:01:18,588 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:01:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:01:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:28.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 05:01:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:28 UTC)" (scheduled at 2025-02-07 05:01:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:01:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:01:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 18, 732859), True, datetime.datetime(2025, 2, 7, 5, 1, 18, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:01:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:01:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:01:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:01:18,968 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:01:19,166 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:01:19,378 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:01:19,602 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:01:19,610 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:01:19,614 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:01:19,617 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:01:19,621 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:01:19,630 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:01:19,633 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:01:19,636 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:01:19,638 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 05:01:20,854 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:01:20,918 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:01:21,387 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:01:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:01:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:03.419110+00:00 (in 39.997752 seconds)
repositorygcworker stdout | 2025-02-07 05:01:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:23 UTC)" (scheduled at 2025-02-07 05:01:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:01:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:01:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:01:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:01:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:40.424250+00:00 (in 16.000678 seconds)
gcworker stdout | 2025-02-07 05:01:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:54 UTC)" (scheduled at 2025-02-07 05:01:24.423055+00:00)
gcworker stdout | 2025-02-07 05:01:24,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 05:01:24,430 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:01:24,430 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:01:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:01:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:01:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:01:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:38.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 05:01:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:38 UTC)" (scheduled at 2025-02-07 05:01:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:01:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:01:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 28, 732872), True, datetime.datetime(2025, 2, 7, 5, 1, 28, 732872), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:01:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:01:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:01:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:01:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:01:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:44.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 05:01:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:44 UTC)" (scheduled at 2025-02-07 05:01:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:01:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:01:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:01:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:01:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:01:29,235 [237] [DEBUG] [app] Starting request: urn:request:577591c0-f3a3-4031-a2b0-6f4353ca3d32 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:29,235 [235] [DEBUG] [app] Starting request: urn:request:a1890d11-fba7-40f0-bf7a-74c51a434c10 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:01:29,238 [239] [DEBUG] [app] Starting request: urn:request:6c439b40-bed8-46cf-a81d-8ba0aa47af23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:29,238 [245] [DEBUG] [app] Starting request: urn:request:b1e7749e-c173-4409-b609-2ea85dd96100 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:29,238 [239] [DEBUG] [app] Ending request: urn:request:6c439b40-bed8-46cf-a81d-8ba0aa47af23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6c439b40-bed8-46cf-a81d-8ba0aa47af23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:01:29,239 [245] [DEBUG] [app] Ending request: urn:request:b1e7749e-c173-4409-b609-2ea85dd96100 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1e7749e-c173-4409-b609-2ea85dd96100', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:01:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:01:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:01:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:29,241 [236] [DEBUG] [app] Starting request: urn:request:ae64174e-8cd5-405e-8f98-caf0ab315d75 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:29,242 [236] [DEBUG] [app] Ending request: urn:request:ae64174e-8cd5-405e-8f98-caf0ab315d75 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae64174e-8cd5-405e-8f98-caf0ab315d75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:01:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:29,242 [235] [DEBUG] [app] Starting request: urn:request:fb243d40-b344-4212-9438-7b4c7bec704a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:29,242 [235] [DEBUG] [app] Ending request: urn:request:fb243d40-b344-4212-9438-7b4c7bec704a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fb243d40-b344-4212-9438-7b4c7bec704a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:01:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:01:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:01:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:01:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:29,254 [235] [DEBUG] [app] Ending request: urn:request:a1890d11-fba7-40f0-bf7a-74c51a434c10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a1890d11-fba7-40f0-bf7a-74c51a434c10', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:29,254 [237] [DEBUG] [app] Ending request: urn:request:577591c0-f3a3-4031-a2b0-6f4353ca3d32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:577591c0-f3a3-4031-a2b0-6f4353ca3d32', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:01:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:01:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:01:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 05:01:30,313 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:01:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:01:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:32.625945+00:00 (in 1.004351 seconds)
namespacegcworker stdout | 2025-02-07 05:01:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:31 UTC)" (scheduled at 2025-02-07 05:01:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:01:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:01:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 31, 621894), True, datetime.datetime(2025, 2, 7, 5, 1, 31, 621894), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:01:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:01:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:01:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:01:32,097 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:01:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:01:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:13.624301+00:00 (in 40.997909 seconds)
namespacegcworker stdout | 2025-02-07 05:01:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:32 UTC)" (scheduled at 2025-02-07 05:01:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:01:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:01:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:01:33,254 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:01:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:01:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:03.919286+00:00 (in 29.999571 seconds)
buildlogsarchiver stdout | 2025-02-07 05:01:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:03 UTC)" (scheduled at 2025-02-07 05:01:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:01:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 1, 33, 919961), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:01:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:01:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:01:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:01:37,549 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:56.713547+00:00 (in 18.997190 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:37 UTC)" (scheduled at 2025-02-07 05:01:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:01:38,593 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:01:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:01:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:48.732147+00:00 (in 9.999574 seconds)
notificationworker stdout | 2025-02-07 05:01:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:48 UTC)" (scheduled at 2025-02-07 05:01:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:01:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:01:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 38, 732840), True, datetime.datetime(2025, 2, 7, 5, 1, 38, 732840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:01:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:01:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:01:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:01:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:01:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:54.423055+00:00 (in 13.998294 seconds)
gcworker stdout | 2025-02-07 05:01:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:10 UTC)" (scheduled at 2025-02-07 05:01:40.424250+00:00)
gcworker stdout | 2025-02-07 05:01:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:01:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904200431, None, 1, 0])
gcworker stdout | 2025-02-07 05:01:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:01:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:01:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:01:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:11.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 05:01:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:11 UTC)" (scheduled at 2025-02-07 05:01:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:01:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900901836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:01:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:01:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:01:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:01:42,504 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:01:43,073 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:01:43,648 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:01:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:01:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:59.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 05:01:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:59 UTC)" (scheduled at 2025-02-07 05:01:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:01:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:01:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:01:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:01:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:01:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:01:44,235 [238] [DEBUG] [app] Starting request: urn:request:f320d290-224b-432b-a0c5-a1150e93ead9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:44,235 [237] [DEBUG] [app] Starting request: urn:request:fab89fdd-23e4-437a-83bb-cb310a209509 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:01:44,238 [246] [DEBUG] [app] Starting request: urn:request:c13f8d48-910f-406b-b5d3-49dd7830647f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:44,238 [239] [DEBUG] [app] Starting request: urn:request:ee6dbb61-48f0-4de1-ab85-4c431384aaaf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:44,238 [246] [DEBUG] [app] Ending request: urn:request:c13f8d48-910f-406b-b5d3-49dd7830647f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c13f8d48-910f-406b-b5d3-49dd7830647f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:01:44,238 [239] [DEBUG] [app] Ending request: urn:request:ee6dbb61-48f0-4de1-ab85-4c431384aaaf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee6dbb61-48f0-4de1-ab85-4c431384aaaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:01:44,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:01:44,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:44,241 [235] [DEBUG] [app] Starting request: urn:request:b2772c72-a1ec-4074-852d-4d7177d029c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:44,241 [235] [DEBUG] [app] Ending request: urn:request:b2772c72-a1ec-4074-852d-4d7177d029c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2772c72-a1ec-4074-852d-4d7177d029c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:01:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:01:44,242 [237] [DEBUG] [app] Starting request: urn:request:6b876ae6-ae93-46bd-8420-850646a168a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:44,242 [237] [DEBUG] [app] Ending request: urn:request:6b876ae6-ae93-46bd-8420-850646a168a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b876ae6-ae93-46bd-8420-850646a168a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:01:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:01:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:01:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:44,253 [237] [DEBUG] [app] Ending request: urn:request:fab89fdd-23e4-437a-83bb-cb310a209509 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fab89fdd-23e4-437a-83bb-cb310a209509', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:44,253 [238] [DEBUG] [app] Ending request: urn:request:f320d290-224b-432b-a0c5-a1150e93ead9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f320d290-224b-432b-a0c5-a1150e93ead9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:01:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:01:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:01:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 05:01:44,970 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:01:45,898 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:01:45,910 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:01:47,078 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:01:47,292 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:01:47,301 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:01:47,304 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:01:47,309 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:01:47,311 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:47,505 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:01:47,953 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:01:47,968 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:01:48,195 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:01:48,317 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:01:48,320 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:01:48,323 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:01:48,618 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:01:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:01:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:50.814677+00:00 (in 2.082085 seconds)
notificationworker stdout | 2025-02-07 05:01:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:58 UTC)" (scheduled at 2025-02-07 05:01:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:01:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:01:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 48, 732871), True, datetime.datetime(2025, 2, 7, 5, 1, 48, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:01:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:01:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:01:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:01:58 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:01:49,004 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:01:49,196 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:01:49,400 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:01:49,610 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:01:49,624 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:01:49,627 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:01:49,630 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:01:49,632 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:01:49,641 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:01:49,644 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:01:49,647 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:01:49,650 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
notificationworker stdout | 2025-02-07 05:01:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:01:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:01:58.732147+00:00 (in 7.917016 seconds)
notificationworker stdout | 2025-02-07 05:01:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:50 UTC)" (scheduled at 2025-02-07 05:01:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:01:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:01:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:01:50,874 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:01:50,955 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:01:51,402 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:01:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:01:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:10.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 05:01:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:24 UTC)" (scheduled at 2025-02-07 05:01:54.423055+00:00)
gcworker stdout | 2025-02-07 05:01:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:01:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:01:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:01:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:56.118855+00:00 (in 59.999483 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:01:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:56 UTC)" (scheduled at 2025-02-07 05:01:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:01:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:01:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:01:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:37.715878+00:00 (in 41.001865 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:56 UTC)" (scheduled at 2025-02-07 05:01:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:01:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:01:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:01:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:08.732147+00:00 (in 9.999571 seconds)
notificationworker stdout | 2025-02-07 05:01:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:08 UTC)" (scheduled at 2025-02-07 05:01:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:01:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:01:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 1, 58, 732852), True, datetime.datetime(2025, 2, 7, 5, 1, 58, 732852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:01:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:01:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:01:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:01:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:01:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:14.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 05:01:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:14 UTC)" (scheduled at 2025-02-07 05:01:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:01:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:01:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:01:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:01:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:01:59,235 [237] [DEBUG] [app] Starting request: urn:request:0394db3a-cc34-4639-bb4a-69c59fd57fb1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:59,235 [238] [DEBUG] [app] Starting request: urn:request:7da5007b-79a9-44ed-80b7-56729bc86765 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:01:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:01:59,238 [245] [DEBUG] [app] Starting request: urn:request:254f774e-12d7-406f-9902-83bbb88600d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:59,238 [239] [DEBUG] [app] Starting request: urn:request:fefd6c43-89d5-46c6-9d78-1738099e1071 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:01:59,239 [245] [DEBUG] [app] Ending request: urn:request:254f774e-12d7-406f-9902-83bbb88600d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:254f774e-12d7-406f-9902-83bbb88600d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:01:59,239 [239] [DEBUG] [app] Ending request: urn:request:fefd6c43-89d5-46c6-9d78-1738099e1071 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fefd6c43-89d5-46c6-9d78-1738099e1071', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:01:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:01:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:01:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:01:59,242 [238] [DEBUG] [app] Starting request: urn:request:1f54ec6c-b118-4ffb-9878-f81d4583e9b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:59,242 [238] [DEBUG] [app] Ending request: urn:request:1f54ec6c-b118-4ffb-9878-f81d4583e9b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1f54ec6c-b118-4ffb-9878-f81d4583e9b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:01:59,242 [235] [DEBUG] [app] Starting request: urn:request:583f3d85-e6eb-4a08-81f8-7bc44dd29b1f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:01:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:01:59,242 [235] [DEBUG] [app] Ending request: urn:request:583f3d85-e6eb-4a08-81f8-7bc44dd29b1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:583f3d85-e6eb-4a08-81f8-7bc44dd29b1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:01:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:01:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:01:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:01:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:01:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:01:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:01:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:01:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:01:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:01:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:01:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:01:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:01:59,254 [238] [DEBUG] [app] Ending request: urn:request:7da5007b-79a9-44ed-80b7-56729bc86765 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7da5007b-79a9-44ed-80b7-56729bc86765', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:59,254 [237] [DEBUG] [app] Ending request: urn:request:0394db3a-cc34-4639-bb4a-69c59fd57fb1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0394db3a-cc34-4639-bb4a-69c59fd57fb1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:01:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:01:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:01:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:01:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:01:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:01:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:01:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:01:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:04.936688+00:00 (in 4.998285 seconds)
exportactionlogsworker stdout | 2025-02-07 05:01:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:59 UTC)" (scheduled at 2025-02-07 05:01:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:01:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:01:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:02:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:02:00,349 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:02:02,128 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:02:03,290 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 05:02:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:02:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:23.420886+00:00 (in 20.001318 seconds)
repositorygcworker stdout | 2025-02-07 05:02:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:03 UTC)" (scheduled at 2025-02-07 05:02:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:02:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:02:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 3, 419836), True, datetime.datetime(2025, 2, 7, 5, 2, 3, 419836), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:02:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:02:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:02:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:02:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:02:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:33.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 05:02:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:33 UTC)" (scheduled at 2025-02-07 05:02:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:02:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 2, 3, 919987), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:02:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:02:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:02:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:02:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:02:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:59.937899+00:00 (in 55.000747 seconds)
exportactionlogsworker stdout | 2025-02-07 05:02:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:04 UTC)" (scheduled at 2025-02-07 05:02:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:02:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:02:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 4, 937424), True, datetime.datetime(2025, 2, 7, 5, 2, 4, 937424), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:02:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:02:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:02:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:02:07,584 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:02:08,629 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:02:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:18.732147+00:00 (in 9.999613 seconds)
notificationworker stdout | 2025-02-07 05:02:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:18 UTC)" (scheduled at 2025-02-07 05:02:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:02:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:02:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 8, 732798), True, datetime.datetime(2025, 2, 7, 5, 2, 8, 732798), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:02:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:02:08,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:02:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:02:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:02:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:24.423055+00:00 (in 13.998374 seconds)
gcworker stdout | 2025-02-07 05:02:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:40 UTC)" (scheduled at 2025-02-07 05:02:10.424250+00:00)
gcworker stdout | 2025-02-07 05:02:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:02:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904230431, None, 1, 0])
gcworker stdout | 2025-02-07 05:02:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:02:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:02:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:02:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:41.831521+00:00 (in 29.999576 seconds)
autopruneworker stdout | 2025-02-07 05:02:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:41 UTC)" (scheduled at 2025-02-07 05:02:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:02:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900931836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:02:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:02:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:02:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:02:12,524 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:02:13,109 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
namespacegcworker stdout | 2025-02-07 05:02:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:02:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:31.621131+00:00 (in 17.996393 seconds)
namespacegcworker stdout | 2025-02-07 05:02:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:07:13 UTC)" (scheduled at 2025-02-07 05:02:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:02:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 2, 13, 624986), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:02:13,652 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 2, 13, 624986), True, datetime.datetime(2025, 2, 7, 5, 2, 13, 624986), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:02:13,661 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 2, 13, 624986), True, datetime.datetime(2025, 2, 7, 5, 2, 13, 624986), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 2, 13, 624986), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:02:13,670 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:02:13,670 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:07:13 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 05:02:13,684 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:02:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:02:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:29.225378+00:00 (in 14.999539 seconds)
quotatotalworker stdout | 2025-02-07 05:02:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:29 UTC)" (scheduled at 2025-02-07 05:02:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:02:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:02:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:02:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:02:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:02:14,235 [235] [DEBUG] [app] Starting request: urn:request:9c310b5f-bc87-4085-853b-b3fe4e432905 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:14,235 [238] [DEBUG] [app] Starting request: urn:request:bd8be7f9-c858-4abc-bad2-9637281762ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:02:14,238 [239] [DEBUG] [app] Starting request: urn:request:4d188063-b5c6-4fca-a104-2b30b5dff441 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:14,238 [246] [DEBUG] [app] Starting request: urn:request:d42c7485-2079-40c6-a3bc-5c1fd16ce4ed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:14,238 [246] [DEBUG] [app] Ending request: urn:request:d42c7485-2079-40c6-a3bc-5c1fd16ce4ed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d42c7485-2079-40c6-a3bc-5c1fd16ce4ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:02:14,238 [239] [DEBUG] [app] Ending request: urn:request:4d188063-b5c6-4fca-a104-2b30b5dff441 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d188063-b5c6-4fca-a104-2b30b5dff441', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:02:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:02:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:02:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:02:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:14,241 [235] [DEBUG] [app] Starting request: urn:request:ec1070a6-966b-4ac3-83ee-540d0456fe24 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:14,241 [237] [DEBUG] [app] Starting request: urn:request:32fee1f4-cd6d-4f95-8178-66810589f5cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:14,241 [235] [DEBUG] [app] Ending request: urn:request:ec1070a6-966b-4ac3-83ee-540d0456fe24 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ec1070a6-966b-4ac3-83ee-540d0456fe24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:02:14,242 [237] [DEBUG] [app] Ending request: urn:request:32fee1f4-cd6d-4f95-8178-66810589f5cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32fee1f4-cd6d-4f95-8178-66810589f5cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:02:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:02:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:02:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:14,253 [238] [DEBUG] [app] Ending request: urn:request:bd8be7f9-c858-4abc-bad2-9637281762ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bd8be7f9-c858-4abc-bad2-9637281762ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:02:14,253 [235] [DEBUG] [app] Ending request: urn:request:9c310b5f-bc87-4085-853b-b3fe4e432905 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c310b5f-bc87-4085-853b-b3fe4e432905', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:02:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:02:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 05:02:15,006 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:02:15,918 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:02:15,947 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:02:17,114 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:02:17,300 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:02:17,310 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:02:17,313 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:02:17,316 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:02:17,319 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:17,518 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:02:17,991 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:02:17,994 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:02:18,231 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:02:18,325 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:02:18,328 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:02:18,331 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:02:18,628 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:02:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:28.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:02:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:28 UTC)" (scheduled at 2025-02-07 05:02:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:02:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:02:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 18, 732840), True, datetime.datetime(2025, 2, 7, 5, 2, 18, 732840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:02:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:02:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:02:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:02:19,036 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:02:19,232 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:02:19,437 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:02:19,623 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:02:19,636 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:02:19,640 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:02:19,643 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:02:19,645 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:02:19,648 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:02:19,653 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:02:19,656 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:02:19,659 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 05:02:20,904 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:02:20,992 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:02:21,437 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:02:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:02:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:03.419110+00:00 (in 39.997752 seconds)
repositorygcworker stdout | 2025-02-07 05:02:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:23 UTC)" (scheduled at 2025-02-07 05:02:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:02:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:02:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:02:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:02:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:40.424250+00:00 (in 16.000736 seconds)
gcworker stdout | 2025-02-07 05:02:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:54 UTC)" (scheduled at 2025-02-07 05:02:24.423055+00:00)
gcworker stdout | 2025-02-07 05:02:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:02:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:02:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:02:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:33.814166+00:00 (in 5.081580 seconds)
notificationworker stdout | 2025-02-07 05:02:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:38 UTC)" (scheduled at 2025-02-07 05:02:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:02:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:02:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 28, 732866), True, datetime.datetime(2025, 2, 7, 5, 2, 28, 732866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:02:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:02:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:02:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:02:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:02:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:44.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 05:02:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:44 UTC)" (scheduled at 2025-02-07 05:02:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:02:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:02:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:02:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:02:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:02:29,235 [237] [DEBUG] [app] Starting request: urn:request:43e49d6e-428c-4892-8e49-dfb26e1355b5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:29,235 [235] [DEBUG] [app] Starting request: urn:request:78f4c942-bdd6-4339-8ac9-b4059c8402e9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:02:29,238 [246] [DEBUG] [app] Starting request: urn:request:e30c5af1-a1e9-40f0-b101-4d256b97d992 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:29,238 [245] [DEBUG] [app] Starting request: urn:request:6cd3fe0d-a431-4f49-bbcc-feaa440bd64f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:29,239 [246] [DEBUG] [app] Ending request: urn:request:e30c5af1-a1e9-40f0-b101-4d256b97d992 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e30c5af1-a1e9-40f0-b101-4d256b97d992', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:02:29,239 [245] [DEBUG] [app] Ending request: urn:request:6cd3fe0d-a431-4f49-bbcc-feaa440bd64f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6cd3fe0d-a431-4f49-bbcc-feaa440bd64f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:02:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:02:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:02:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:02:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:29,241 [235] [DEBUG] [app] Starting request: urn:request:f6bbcf92-315c-47ba-998e-0e833c024fd7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:29,241 [235] [DEBUG] [app] Ending request: urn:request:f6bbcf92-315c-47ba-998e-0e833c024fd7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6bbcf92-315c-47ba-998e-0e833c024fd7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:02:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:02:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:29,243 [238] [DEBUG] [app] Starting request: urn:request:f6262f3f-f5d0-4dee-9b4e-cb619fed81e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:29,243 [238] [DEBUG] [app] Ending request: urn:request:f6262f3f-f5d0-4dee-9b4e-cb619fed81e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6262f3f-f5d0-4dee-9b4e-cb619fed81e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:02:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:02:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:29,253 [237] [DEBUG] [app] Ending request: urn:request:43e49d6e-428c-4892-8e49-dfb26e1355b5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43e49d6e-428c-4892-8e49-dfb26e1355b5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:02:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 05:02:29,254 [235] [DEBUG] [app] Ending request: urn:request:78f4c942-bdd6-4339-8ac9-b4059c8402e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78f4c942-bdd6-4339-8ac9-b4059c8402e9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:02:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 05:02:30,373 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:02:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:02:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:32.625945+00:00 (in 1.004343 seconds)
namespacegcworker stdout | 2025-02-07 05:02:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:31 UTC)" (scheduled at 2025-02-07 05:02:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:02:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:02:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 31, 621928), True, datetime.datetime(2025, 2, 7, 5, 2, 31, 621928), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:02:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:02:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:02:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:02:32,147 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:02:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:02:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:31.621131+00:00 (in 58.994715 seconds)
namespacegcworker stdout | 2025-02-07 05:02:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:32 UTC)" (scheduled at 2025-02-07 05:02:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:02:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:02:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:02:33,318 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:02:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:38.732147+00:00 (in 4.917530 seconds)
notificationworker stdout | 2025-02-07 05:02:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:07:33 UTC)" (scheduled at 2025-02-07 05:02:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:02:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 2, 33, 814877), 'notification/%'])
notificationworker stdout | 2025-02-07 05:02:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 2, 33, 814877), True, datetime.datetime(2025, 2, 7, 5, 2, 33, 814877), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:02:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 2, 33, 814877), True, datetime.datetime(2025, 2, 7, 5, 2, 33, 814877), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 2, 33, 814877), 'notification/%'])
notificationworker stdout | 2025-02-07 05:02:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:02:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:07:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:02:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:02:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:03.919286+00:00 (in 29.999510 seconds)
buildlogsarchiver stdout | 2025-02-07 05:02:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:03 UTC)" (scheduled at 2025-02-07 05:02:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:02:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 2, 33, 920056), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:02:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:02:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:02:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:02:37,618 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:56.713547+00:00 (in 18.997162 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:37 UTC)" (scheduled at 2025-02-07 05:02:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:02:38,665 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:02:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:48.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 05:02:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:48 UTC)" (scheduled at 2025-02-07 05:02:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:02:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:02:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 38, 732871), True, datetime.datetime(2025, 2, 7, 5, 2, 38, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:02:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:02:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:02:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:02:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:02:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:54.423055+00:00 (in 13.998359 seconds)
gcworker stdout | 2025-02-07 05:02:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:10 UTC)" (scheduled at 2025-02-07 05:02:40.424250+00:00)
gcworker stdout | 2025-02-07 05:02:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:02:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904260431, None, 1, 0])
gcworker stdout | 2025-02-07 05:02:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:02:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:02:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:02:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:11.831521+00:00 (in 29.999563 seconds)
autopruneworker stdout | 2025-02-07 05:02:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:11 UTC)" (scheduled at 2025-02-07 05:02:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:02:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900961836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:02:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:02:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:02:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:02:42,534 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:02:43,129 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:02:43,706 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:02:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:02:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:59.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 05:02:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:59 UTC)" (scheduled at 2025-02-07 05:02:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:02:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:02:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:02:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:02:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:02:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:02:44,236 [238] [DEBUG] [app] Starting request: urn:request:b6cc2496-f8b6-4849-ab1c-8cbf029a9737 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:44,236 [235] [DEBUG] [app] Starting request: urn:request:86701657-ff4d-488b-8d7c-1ab347871b7c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:02:44,239 [239] [DEBUG] [app] Starting request: urn:request:27178db5-c945-4ce3-b272-74951565cf43 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:44,239 [246] [DEBUG] [app] Starting request: urn:request:66e9bbb7-2a0c-4835-a9aa-68596937ac58 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:44,239 [239] [DEBUG] [app] Ending request: urn:request:27178db5-c945-4ce3-b272-74951565cf43 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27178db5-c945-4ce3-b272-74951565cf43', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:02:44,239 [246] [DEBUG] [app] Ending request: urn:request:66e9bbb7-2a0c-4835-a9aa-68596937ac58 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:66e9bbb7-2a0c-4835-a9aa-68596937ac58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:02:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:02:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:02:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:44,242 [235] [DEBUG] [app] Starting request: urn:request:eac2b282-ac7b-4557-a8d0-f5bb16aa102f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:44,242 [235] [DEBUG] [app] Ending request: urn:request:eac2b282-ac7b-4557-a8d0-f5bb16aa102f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eac2b282-ac7b-4557-a8d0-f5bb16aa102f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:02:44,242 [237] [DEBUG] [app] Starting request: urn:request:3f5d12a2-ada0-46ad-9bf3-3c3e56f06169 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:02:44,243 [237] [DEBUG] [app] Ending request: urn:request:3f5d12a2-ada0-46ad-9bf3-3c3e56f06169 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f5d12a2-ada0-46ad-9bf3-3c3e56f06169', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:02:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:02:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:44,254 [238] [DEBUG] [app] Ending request: urn:request:b6cc2496-f8b6-4849-ab1c-8cbf029a9737 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6cc2496-f8b6-4849-ab1c-8cbf029a9737', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:44,254 [235] [DEBUG] [app] Ending request: urn:request:86701657-ff4d-488b-8d7c-1ab347871b7c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:86701657-ff4d-488b-8d7c-1ab347871b7c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:02:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:02:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:02:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 05:02:45,026 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:02:45,953 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:02:45,983 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:02:47,151 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:02:47,307 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:02:47,317 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:02:47,321 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:02:47,324 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:02:47,327 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:47,554 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:02:48,012 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:02:48,015 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:02:48,268 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:02:48,333 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:02:48,336 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:02:48,338 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:02:48,639 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:02:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:50.814677+00:00 (in 2.082107 seconds)
notificationworker stdout | 2025-02-07 05:02:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:58 UTC)" (scheduled at 2025-02-07 05:02:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:02:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:02:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 48, 732850), True, datetime.datetime(2025, 2, 7, 5, 2, 48, 732850), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:02:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:02:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:02:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:02:58 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:02:49,072 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:02:49,258 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:02:49,472 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:02:49,635 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:02:49,644 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:02:49,648 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:02:49,651 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:02:49,655 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:02:49,659 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:02:49,661 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:02:49,664 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:02:49,666 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
notificationworker stdout | 2025-02-07 05:02:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:02:58.732147+00:00 (in 7.917009 seconds)
notificationworker stdout | 2025-02-07 05:02:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:50 UTC)" (scheduled at 2025-02-07 05:02:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:02:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:02:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:02:50,941 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:02:51,027 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:02:51,472 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:02:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:02:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:10.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 05:02:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:24 UTC)" (scheduled at 2025-02-07 05:02:54.423055+00:00)
gcworker stdout | 2025-02-07 05:02:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:02:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:02:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:02:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:56.118855+00:00 (in 59.999488 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:02:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:56 UTC)" (scheduled at 2025-02-07 05:02:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:02:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:02:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:02:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:37.715878+00:00 (in 41.001876 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:56 UTC)" (scheduled at 2025-02-07 05:02:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:02:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:02:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:02:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:08.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 05:02:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:08 UTC)" (scheduled at 2025-02-07 05:02:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:02:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:02:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 2, 58, 732898), True, datetime.datetime(2025, 2, 7, 5, 2, 58, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:02:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:02:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:02:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:02:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:02:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:14.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 05:02:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:14 UTC)" (scheduled at 2025-02-07 05:02:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:02:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:02:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:02:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:02:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:02:59,236 [235] [DEBUG] [app] Starting request: urn:request:cd3c958f-ca92-4a77-9ddb-897d0a2d789f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:59,236 [237] [DEBUG] [app] Starting request: urn:request:d9540fea-4f78-457f-a658-5a2f7aa35b0f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:02:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:02:59,239 [246] [DEBUG] [app] Starting request: urn:request:1b18f3ca-6804-4604-9d49-3a943e39c73a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:59,239 [245] [DEBUG] [app] Starting request: urn:request:daee8c1e-9a85-43b6-b4e3-a41a2baf8ed7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:02:59,240 [245] [DEBUG] [app] Ending request: urn:request:daee8c1e-9a85-43b6-b4e3-a41a2baf8ed7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:daee8c1e-9a85-43b6-b4e3-a41a2baf8ed7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:02:59,240 [246] [DEBUG] [app] Ending request: urn:request:1b18f3ca-6804-4604-9d49-3a943e39c73a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b18f3ca-6804-4604-9d49-3a943e39c73a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:02:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:02:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:02:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:59,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:02:59,243 [235] [DEBUG] [app] Starting request: urn:request:f3a71cad-197e-4e08-82df-01accd084d3f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:59,244 [235] [DEBUG] [app] Ending request: urn:request:f3a71cad-197e-4e08-82df-01accd084d3f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3a71cad-197e-4e08-82df-01accd084d3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:02:59,244 [238] [DEBUG] [app] Starting request: urn:request:404ec9f0-af1a-4812-a1cd-f4778a03922a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:02:59,244 [238] [DEBUG] [app] Ending request: urn:request:404ec9f0-af1a-4812-a1cd-f4778a03922a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:404ec9f0-af1a-4812-a1cd-f4778a03922a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:02:59,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:02:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:02:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:02:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:02:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:02:59,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:59,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:59,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:02:59,246 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:02:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:59,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:02:59,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:02:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:59,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:02:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:59,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:02:59,256 [235] [DEBUG] [app] Ending request: urn:request:cd3c958f-ca92-4a77-9ddb-897d0a2d789f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd3c958f-ca92-4a77-9ddb-897d0a2d789f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:02:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:02:59,257 [237] [DEBUG] [app] Ending request: urn:request:d9540fea-4f78-457f-a658-5a2f7aa35b0f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9540fea-4f78-457f-a658-5a2f7aa35b0f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:02:59,257 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:02:59,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:02:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:02:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
exportactionlogsworker stdout | 2025-02-07 05:02:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:02:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:04.936688+00:00 (in 4.998200 seconds)
exportactionlogsworker stdout | 2025-02-07 05:02:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:59 UTC)" (scheduled at 2025-02-07 05:02:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:02:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:02:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:03:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:03:00,402 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:03:02,156 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:03:03,344 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 05:03:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:03:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:23.420886+00:00 (in 20.001294 seconds)
repositorygcworker stdout | 2025-02-07 05:03:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:03 UTC)" (scheduled at 2025-02-07 05:03:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:03:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:03:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 3, 419864), True, datetime.datetime(2025, 2, 7, 5, 3, 3, 419864), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:03:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:03:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:03:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:03:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:03:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:33.919286+00:00 (in 29.999507 seconds)
buildlogsarchiver stdout | 2025-02-07 05:03:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:33 UTC)" (scheduled at 2025-02-07 05:03:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:03:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 3, 3, 920027), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:03:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:03:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:03:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:03:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:03:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:59.937899+00:00 (in 55.000767 seconds)
exportactionlogsworker stdout | 2025-02-07 05:03:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:04 UTC)" (scheduled at 2025-02-07 05:03:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:03:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:03:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 4, 937358), True, datetime.datetime(2025, 2, 7, 5, 3, 4, 937358), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:03:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:03:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:03:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:03:07,640 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:03:08,686 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:03:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:03:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:18.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 05:03:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:18 UTC)" (scheduled at 2025-02-07 05:03:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:03:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:03:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 8, 732926), True, datetime.datetime(2025, 2, 7, 5, 3, 8, 732926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:03:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:03:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:03:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:03:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:03:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:24.423055+00:00 (in 13.998364 seconds)
gcworker stdout | 2025-02-07 05:03:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:40 UTC)" (scheduled at 2025-02-07 05:03:10.424250+00:00)
gcworker stdout | 2025-02-07 05:03:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:03:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904290431, None, 1, 0])
gcworker stdout | 2025-02-07 05:03:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:03:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:03:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:03:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:41.831521+00:00 (in 29.999543 seconds)
autopruneworker stdout | 2025-02-07 05:03:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:41 UTC)" (scheduled at 2025-02-07 05:03:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:03:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738900991836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:03:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:03:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:03:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:03:12,559 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:03:13,141 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:03:13,738 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:03:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:03:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:29.225378+00:00 (in 14.999524 seconds)
quotatotalworker stdout | 2025-02-07 05:03:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:29 UTC)" (scheduled at 2025-02-07 05:03:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:03:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:03:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:03:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:03:14,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:03:14,235 [237] [DEBUG] [app] Starting request: urn:request:13407b29-8088-485c-bcfc-d26dd91e50dc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:14,235 [238] [DEBUG] [app] Starting request: urn:request:3002c77f-b8e2-4611-bc3d-45bc7801e278 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:03:14,238 [239] [DEBUG] [app] Starting request: urn:request:165c3938-2000-4f50-bbda-e48362d32148 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:14,238 [244] [DEBUG] [app] Starting request: urn:request:3b6102b1-8957-43aa-9dd6-af219dff3a63 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:14,238 [239] [DEBUG] [app] Ending request: urn:request:165c3938-2000-4f50-bbda-e48362d32148 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:165c3938-2000-4f50-bbda-e48362d32148', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:03:14,238 [244] [DEBUG] [app] Ending request: urn:request:3b6102b1-8957-43aa-9dd6-af219dff3a63 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b6102b1-8957-43aa-9dd6-af219dff3a63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:03:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:03:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:03:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:14,241 [235] [DEBUG] [app] Starting request: urn:request:7c484415-c1f7-46ba-891b-8ed14b1675e0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:03:14,242 [235] [DEBUG] [app] Ending request: urn:request:7c484415-c1f7-46ba-891b-8ed14b1675e0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c484415-c1f7-46ba-891b-8ed14b1675e0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:14,242 [238] [DEBUG] [app] Starting request: urn:request:e34fdc5c-a808-4545-bb8c-fdbedb93802b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:03:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:14,242 [238] [DEBUG] [app] Ending request: urn:request:e34fdc5c-a808-4545-bb8c-fdbedb93802b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e34fdc5c-a808-4545-bb8c-fdbedb93802b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:03:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:03:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:03:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:14,254 [238] [DEBUG] [app] Ending request: urn:request:3002c77f-b8e2-4611-bc3d-45bc7801e278 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3002c77f-b8e2-4611-bc3d-45bc7801e278', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:03:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:03:14,254 [237] [DEBUG] [app] Ending request: urn:request:13407b29-8088-485c-bcfc-d26dd91e50dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13407b29-8088-485c-bcfc-d26dd91e50dc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:03:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 05:03:15,062 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:03:15,969 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:03:16,019 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:03:17,186 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:03:17,315 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:03:17,324 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:03:17,329 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:03:17,332 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:03:17,335 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:17,590 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:03:18,023 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:03:18,050 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:03:18,305 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:03:18,341 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:03:18,344 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:03:18,347 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:03:18,650 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:03:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:03:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:28.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 05:03:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:28 UTC)" (scheduled at 2025-02-07 05:03:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:03:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:03:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 18, 732947), True, datetime.datetime(2025, 2, 7, 5, 3, 18, 732947), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:03:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:03:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:03:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:03:19,098 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:03:19,274 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:03:19,508 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:03:19,647 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:03:19,653 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:03:19,658 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:03:19,661 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:03:19,665 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:03:19,668 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:03:19,671 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:03:19,674 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:03:19,677 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
buildlogsarchiver stdout | 2025-02-07 05:03:20,970 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:03:21,064 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:03:21,491 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:03:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:03:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:32.420707+00:00 (in 8.999365 seconds)
repositorygcworker stdout | 2025-02-07 05:03:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:23 UTC)" (scheduled at 2025-02-07 05:03:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:03:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:03:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:03:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:03:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:40.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 05:03:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:54 UTC)" (scheduled at 2025-02-07 05:03:24.423055+00:00)
gcworker stdout | 2025-02-07 05:03:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:03:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:03:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:03:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:03:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:38.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 05:03:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:38 UTC)" (scheduled at 2025-02-07 05:03:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:03:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:03:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 28, 732912), True, datetime.datetime(2025, 2, 7, 5, 3, 28, 732912), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:03:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:03:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:03:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:03:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:03:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:44.225378+00:00 (in 14.999539 seconds)
quotatotalworker stdout | 2025-02-07 05:03:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:44 UTC)" (scheduled at 2025-02-07 05:03:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:03:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:03:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:03:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:03:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:03:29,235 [236] [DEBUG] [app] Starting request: urn:request:52be549c-9bf8-4f16-b50b-7c9074a31094 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:29,235 [235] [DEBUG] [app] Starting request: urn:request:0820eb5e-e507-49a2-b86e-251b4807638b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:03:29,239 [245] [DEBUG] [app] Starting request: urn:request:5f6b2bb8-f283-4298-9c92-99b6cdf9b685 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:29,239 [239] [DEBUG] [app] Starting request: urn:request:555e6e1c-a113-4f04-a160-146a0df1e75a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:29,240 [245] [DEBUG] [app] Ending request: urn:request:5f6b2bb8-f283-4298-9c92-99b6cdf9b685 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f6b2bb8-f283-4298-9c92-99b6cdf9b685', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:03:29,240 [239] [DEBUG] [app] Ending request: urn:request:555e6e1c-a113-4f04-a160-146a0df1e75a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:555e6e1c-a113-4f04-a160-146a0df1e75a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:03:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:03:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:29,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:29,243 [237] [DEBUG] [app] Starting request: urn:request:ff8f3919-c886-4836-b948-7ebfa92f89d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:03:29,243 [238] [DEBUG] [app] Starting request: urn:request:538f5ac2-7a60-4394-bda7-3c28a44b0040 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:03:29,244 [237] [DEBUG] [app] Ending request: urn:request:ff8f3919-c886-4836-b948-7ebfa92f89d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff8f3919-c886-4836-b948-7ebfa92f89d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:03:29,244 [238] [DEBUG] [app] Ending request: urn:request:538f5ac2-7a60-4394-bda7-3c28a44b0040 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:538f5ac2-7a60-4394-bda7-3c28a44b0040', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:29,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:29,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:03:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:03:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:29,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:29,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:29,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:29,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:29,255 [236] [DEBUG] [app] Ending request: urn:request:52be549c-9bf8-4f16-b50b-7c9074a31094 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:52be549c-9bf8-4f16-b50b-7c9074a31094', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:29,255 [235] [DEBUG] [app] Ending request: urn:request:0820eb5e-e507-49a2-b86e-251b4807638b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0820eb5e-e507-49a2-b86e-251b4807638b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:03:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:03:29,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:03:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.021)
storagereplication stdout | 2025-02-07 05:03:30,436 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:03:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:03:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:32.625945+00:00 (in 1.004323 seconds)
namespacegcworker stdout | 2025-02-07 05:03:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:31 UTC)" (scheduled at 2025-02-07 05:03:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:03:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:03:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 31, 621921), True, datetime.datetime(2025, 2, 7, 5, 3, 31, 621921), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:03:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:03:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:03:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:03:32,186 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:03:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:03:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:03.419110+00:00 (in 30.997951 seconds)
repositorygcworker stdout | 2025-02-07 05:03:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:08:32 UTC)" (scheduled at 2025-02-07 05:03:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:03:32,422 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 3, 32, 421464), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:03:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 3, 32, 421464), True, datetime.datetime(2025, 2, 7, 5, 3, 32, 421464), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:03:32,431 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 3, 32, 421464), True, datetime.datetime(2025, 2, 7, 5, 3, 32, 421464), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 3, 32, 421464), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:03:32,432 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:03:32,432 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:08:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:03:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:03:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:31.621131+00:00 (in 58.994708 seconds)
namespacegcworker stdout | 2025-02-07 05:03:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:32 UTC)" (scheduled at 2025-02-07 05:03:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:03:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:03:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:03:33,380 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:03:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:03:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:03.919286+00:00 (in 29.999528 seconds)
buildlogsarchiver stdout | 2025-02-07 05:03:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:03 UTC)" (scheduled at 2025-02-07 05:03:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:03:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 3, 33, 920009), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:03:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:03:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:03:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:03 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:03:37,677 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:56.713547+00:00 (in 18.997201 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:37 UTC)" (scheduled at 2025-02-07 05:03:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:37 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:03:38,711 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
notificationworker stdout | 2025-02-07 05:03:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:03:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:48.732147+00:00 (in 9.999584 seconds)
notificationworker stdout | 2025-02-07 05:03:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:48 UTC)" (scheduled at 2025-02-07 05:03:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:03:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:03:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 38, 732831), True, datetime.datetime(2025, 2, 7, 5, 3, 38, 732831), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:03:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:03:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:03:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:03:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:03:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:54.423055+00:00 (in 13.998376 seconds)
gcworker stdout | 2025-02-07 05:03:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:10 UTC)" (scheduled at 2025-02-07 05:03:40.424250+00:00)
gcworker stdout | 2025-02-07 05:03:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:03:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904320431, None, 1, 0])
gcworker stdout | 2025-02-07 05:03:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:03:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:03:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:03:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:11.831521+00:00 (in 29.999517 seconds)
autopruneworker stdout | 2025-02-07 05:03:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:11 UTC)" (scheduled at 2025-02-07 05:03:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:03:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901021836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:03:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:03:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:03:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:03:42,595 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:03:43,177 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:03:43,774 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:03:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:03:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:59.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 05:03:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:59 UTC)" (scheduled at 2025-02-07 05:03:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:03:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:03:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:03:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:03:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:03:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:03:44,235 [235] [DEBUG] [app] Starting request: urn:request:155338c6-6bdb-4dda-bbeb-a4639c4f65ef (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:44,235 [238] [DEBUG] [app] Starting request: urn:request:a02b976b-8c5e-4b1f-8ab7-98d65bfb8cc6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:03:44,238 [246] [DEBUG] [app] Starting request: urn:request:7a175775-2205-4187-b1d5-91d56be5c1d3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:44,239 [239] [DEBUG] [app] Starting request: urn:request:ad3f347b-06d5-4b3f-8cd9-6313c70119a7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:44,239 [246] [DEBUG] [app] Ending request: urn:request:7a175775-2205-4187-b1d5-91d56be5c1d3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7a175775-2205-4187-b1d5-91d56be5c1d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:03:44,239 [239] [DEBUG] [app] Ending request: urn:request:ad3f347b-06d5-4b3f-8cd9-6313c70119a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ad3f347b-06d5-4b3f-8cd9-6313c70119a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:03:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:03:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:03:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:44,242 [237] [DEBUG] [app] Starting request: urn:request:95004bd1-3a11-4957-9d10-84d4b43e649e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:03:44,242 [237] [DEBUG] [app] Ending request: urn:request:95004bd1-3a11-4957-9d10-84d4b43e649e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95004bd1-3a11-4957-9d10-84d4b43e649e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:03:44,242 [236] [DEBUG] [app] Starting request: urn:request:3a74c204-8cb0-407d-8d58-e8d15caa3291 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:44,243 [236] [DEBUG] [app] Ending request: urn:request:3a74c204-8cb0-407d-8d58-e8d15caa3291 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3a74c204-8cb0-407d-8d58-e8d15caa3291', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:03:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:03:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:44,253 [238] [DEBUG] [app] Ending request: urn:request:a02b976b-8c5e-4b1f-8ab7-98d65bfb8cc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a02b976b-8c5e-4b1f-8ab7-98d65bfb8cc6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:44,253 [235] [DEBUG] [app] Ending request: urn:request:155338c6-6bdb-4dda-bbeb-a4639c4f65ef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:155338c6-6bdb-4dda-bbeb-a4639c4f65ef', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:03:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:03:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:03:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
expiredappspecifictokenworker stdout | 2025-02-07 05:03:45,073 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:03:45,981 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:03:46,055 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:03:47,216 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:03:47,322 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:03:47,331 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:03:47,336 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:03:47,339 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:03:47,342 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:47,627 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:03:48,059 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:03:48,086 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:03:48,336 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:03:48,348 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:03:48,351 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:03:48,354 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:03:48,686 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:03:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:03:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:50.814677+00:00 (in 2.082104 seconds)
notificationworker stdout | 2025-02-07 05:03:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:58 UTC)" (scheduled at 2025-02-07 05:03:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:03:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:03:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 48, 732848), True, datetime.datetime(2025, 2, 7, 5, 3, 48, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:03:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:03:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:03:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:03:58 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:03:49,135 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:03:49,286 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:03:49,545 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:03:49,658 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:03:49,661 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:03:49,668 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:03:49,671 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:03:49,674 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:03:49,677 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:03:49,680 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:03:49,683 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:03:49,687 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 05:03:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:03:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:03:58.732147+00:00 (in 7.917044 seconds)
notificationworker stdout | 2025-02-07 05:03:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:50 UTC)" (scheduled at 2025-02-07 05:03:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:03:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:03:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:03:50,985 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:03:51,095 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:03:51,505 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:03:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:03:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:10.424250+00:00 (in 16.000671 seconds)
gcworker stdout | 2025-02-07 05:03:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:24 UTC)" (scheduled at 2025-02-07 05:03:54.423055+00:00)
gcworker stdout | 2025-02-07 05:03:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:03:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:03:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:03:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:56.118855+00:00 (in 59.999506 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:03:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:56 UTC)" (scheduled at 2025-02-07 05:03:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:03:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:03:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:03:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:37.715878+00:00 (in 41.001884 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:56 UTC)" (scheduled at 2025-02-07 05:03:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:03:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:03:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:03:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:08.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:03:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:08 UTC)" (scheduled at 2025-02-07 05:03:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:03:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:03:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 3, 58, 732853), True, datetime.datetime(2025, 2, 7, 5, 3, 58, 732853), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:03:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:03:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:03:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:03:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:03:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:14.225378+00:00 (in 14.999510 seconds)
quotatotalworker stdout | 2025-02-07 05:03:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:14 UTC)" (scheduled at 2025-02-07 05:03:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:03:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:03:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:03:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:03:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:03:59,235 [235] [DEBUG] [app] Starting request: urn:request:c9f51650-9e56-4255-a88e-c88f6662fa1d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:59,235 [238] [DEBUG] [app] Starting request: urn:request:7fd2acaf-33bd-4e69-8ecd-3aa17dbfb101 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:03:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:03:59,238 [239] [DEBUG] [app] Starting request: urn:request:5bd505d4-c279-4cf8-828a-0fdffbc95d4c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:59,238 [246] [DEBUG] [app] Starting request: urn:request:c2e99194-bf95-4044-8e83-ee8292c16413 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:03:59,238 [239] [DEBUG] [app] Ending request: urn:request:5bd505d4-c279-4cf8-828a-0fdffbc95d4c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5bd505d4-c279-4cf8-828a-0fdffbc95d4c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:03:59,239 [246] [DEBUG] [app] Ending request: urn:request:c2e99194-bf95-4044-8e83-ee8292c16413 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2e99194-bf95-4044-8e83-ee8292c16413', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:03:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:03:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:03:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:03:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:03:59,241 [238] [DEBUG] [app] Starting request: urn:request:b6af5de9-bf93-4781-acc0-576af0d7e34a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:03:59,241 [238] [DEBUG] [app] Ending request: urn:request:b6af5de9-bf93-4781-acc0-576af0d7e34a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6af5de9-bf93-4781-acc0-576af0d7e34a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:03:59,241 [237] [DEBUG] [app] Starting request: urn:request:5be2708a-ff04-495c-9a41-f62d20ce5923 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:03:59,242 [237] [DEBUG] [app] Ending request: urn:request:5be2708a-ff04-495c-9a41-f62d20ce5923 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5be2708a-ff04-495c-9a41-f62d20ce5923', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:03:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:03:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:03:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:03:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:03:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:03:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:03:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:03:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:03:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:03:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:03:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:03:59,253 [238] [DEBUG] [app] Ending request: urn:request:7fd2acaf-33bd-4e69-8ecd-3aa17dbfb101 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7fd2acaf-33bd-4e69-8ecd-3aa17dbfb101', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:59,253 [235] [DEBUG] [app] Ending request: urn:request:c9f51650-9e56-4255-a88e-c88f6662fa1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9f51650-9e56-4255-a88e-c88f6662fa1d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:03:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:03:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:03:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:03:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:03:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:03:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:03:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:03:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:04.936688+00:00 (in 4.998322 seconds)
exportactionlogsworker stdout | 2025-02-07 05:03:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:59 UTC)" (scheduled at 2025-02-07 05:03:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:03:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:03:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:04:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:04:00,471 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:04:02,222 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:04:03,387 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
repositorygcworker stdout | 2025-02-07 05:04:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:04:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:23.420886+00:00 (in 20.001289 seconds)
repositorygcworker stdout | 2025-02-07 05:04:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:03 UTC)" (scheduled at 2025-02-07 05:04:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:04:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:04:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 3, 419887), True, datetime.datetime(2025, 2, 7, 5, 4, 3, 419887), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:04:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:04:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:04:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:04:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:04:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:33.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 05:04:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:33 UTC)" (scheduled at 2025-02-07 05:04:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:04:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 4, 3, 920008), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:04:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:04:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:04:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:04:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:04:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:52.937648+00:00 (in 48.000525 seconds)
exportactionlogsworker stdout | 2025-02-07 05:04:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:04 UTC)" (scheduled at 2025-02-07 05:04:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:04:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:04:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 4, 937432), True, datetime.datetime(2025, 2, 7, 5, 4, 4, 937432), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:04:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:04:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:04:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:04:07,714 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:04:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:04:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:18.732147+00:00 (in 9.999587 seconds)
notificationworker stdout | 2025-02-07 05:04:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:18 UTC)" (scheduled at 2025-02-07 05:04:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:04:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:04:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 8, 732851), True, datetime.datetime(2025, 2, 7, 5, 4, 8, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:04:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:04:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:04:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:04:08,748 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:04:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:04:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:24.423055+00:00 (in 13.998339 seconds)
gcworker stdout | 2025-02-07 05:04:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:40 UTC)" (scheduled at 2025-02-07 05:04:10.424250+00:00)
gcworker stdout | 2025-02-07 05:04:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:04:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904350431, None, 1, 0])
gcworker stdout | 2025-02-07 05:04:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:04:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:04:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:04:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:41.831521+00:00 (in 29.999507 seconds)
autopruneworker stdout | 2025-02-07 05:04:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:41 UTC)" (scheduled at 2025-02-07 05:04:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:04:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901051836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:04:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:04:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:04:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:04:12,631 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:04:13,213 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:04:13,795 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:04:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:04:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:29.225378+00:00 (in 14.999520 seconds)
quotatotalworker stdout | 2025-02-07 05:04:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:29 UTC)" (scheduled at 2025-02-07 05:04:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:04:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:04:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:04:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:04:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:04:14,235 [235] [DEBUG] [app] Starting request: urn:request:7506815e-7abe-4ce1-9599-a29a02badbe2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:14,235 [237] [DEBUG] [app] Starting request: urn:request:b4233d51-a897-46a4-a7b9-28f6d5fafa1c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:04:14,238 [239] [DEBUG] [app] Starting request: urn:request:69763f94-efa6-471c-8696-460bbdcc8f26 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:14,238 [244] [DEBUG] [app] Starting request: urn:request:65126199-a7c9-4067-a2af-469add1be402 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:14,238 [239] [DEBUG] [app] Ending request: urn:request:69763f94-efa6-471c-8696-460bbdcc8f26 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:69763f94-efa6-471c-8696-460bbdcc8f26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:04:14,238 [244] [DEBUG] [app] Ending request: urn:request:65126199-a7c9-4067-a2af-469add1be402 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:65126199-a7c9-4067-a2af-469add1be402', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:04:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:04:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:04:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:14,241 [237] [DEBUG] [app] Starting request: urn:request:032d0a05-8942-494d-a42d-400f1d45c1e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:14,241 [238] [DEBUG] [app] Starting request: urn:request:889d8d96-b407-4b78-9203-a67d926227ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:14,241 [237] [DEBUG] [app] Ending request: urn:request:032d0a05-8942-494d-a42d-400f1d45c1e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:032d0a05-8942-494d-a42d-400f1d45c1e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:04:14,241 [238] [DEBUG] [app] Ending request: urn:request:889d8d96-b407-4b78-9203-a67d926227ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:889d8d96-b407-4b78-9203-a67d926227ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:04:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:04:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:04:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:14,253 [235] [DEBUG] [app] Ending request: urn:request:7506815e-7abe-4ce1-9599-a29a02badbe2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7506815e-7abe-4ce1-9599-a29a02badbe2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:14,253 [237] [DEBUG] [app] Ending request: urn:request:b4233d51-a897-46a4-a7b9-28f6d5fafa1c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b4233d51-a897-46a4-a7b9-28f6d5fafa1c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:04:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 05:04:15,106 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:04:16,012 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:04:16,092 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:04:17,252 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:04:17,329 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:04:17,339 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:04:17,345 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:04:17,349 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:04:17,351 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:17,658 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:04:18,095 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:04:18,099 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:04:18,354 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:04:18,359 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:04:18,361 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:04:18,364 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
repositorygcworker stdout | 2025-02-07 05:04:18,723 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:04:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:04:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:28.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:04:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:28 UTC)" (scheduled at 2025-02-07 05:04:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:04:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:04:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 18, 732858), True, datetime.datetime(2025, 2, 7, 5, 4, 18, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:04:18,758 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:04:18,758 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:04:18,758 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:28 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:04:19,155 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:04:19,318 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:04:19,582 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:04:19,668 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:04:19,671 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:04:19,675 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:04:19,678 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:04:19,680 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:04:19,687 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:04:19,693 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:04:19,696 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:04:19,698 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
buildlogsarchiver stdout | 2025-02-07 05:04:20,993 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:04:21,114 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:04:21,542 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:04:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:04:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:03.419110+00:00 (in 39.997745 seconds)
repositorygcworker stdout | 2025-02-07 05:04:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:23 UTC)" (scheduled at 2025-02-07 05:04:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:04:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:04:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:04:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:04:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:40.424250+00:00 (in 16.000756 seconds)
gcworker stdout | 2025-02-07 05:04:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:54 UTC)" (scheduled at 2025-02-07 05:04:24.423055+00:00)
gcworker stdout | 2025-02-07 05:04:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:04:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:04:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:04:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:04:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:38.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:04:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:38 UTC)" (scheduled at 2025-02-07 05:04:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:04:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:04:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 28, 732869), True, datetime.datetime(2025, 2, 7, 5, 4, 28, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:04:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:04:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:04:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:04:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:04:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:44.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 05:04:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:44 UTC)" (scheduled at 2025-02-07 05:04:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:04:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:04:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:04:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:04:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:04:29,236 [235] [DEBUG] [app] Starting request: urn:request:bea4f100-486f-4522-8948-dd5443a852af (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:29,236 [238] [DEBUG] [app] Starting request: urn:request:dc6ba880-40d3-4f35-8b21-f03b4d644543 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:04:29,240 [245] [DEBUG] [app] Starting request: urn:request:e9a4ad3a-ca13-421f-88aa-6f28d265fb5d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:29,240 [239] [DEBUG] [app] Starting request: urn:request:a9b32436-e1ae-4b07-bf18-b0fdadf3c7a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:29,240 [239] [DEBUG] [app] Ending request: urn:request:a9b32436-e1ae-4b07-bf18-b0fdadf3c7a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9b32436-e1ae-4b07-bf18-b0fdadf3c7a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:04:29,240 [245] [DEBUG] [app] Ending request: urn:request:e9a4ad3a-ca13-421f-88aa-6f28d265fb5d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e9a4ad3a-ca13-421f-88aa-6f28d265fb5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:04:29,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:04:29,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:04:29,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:04:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:29,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:29,244 [237] [DEBUG] [app] Starting request: urn:request:a6af3835-6c05-4fc4-9423-3a2ff9c05b9a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:29,244 [236] [DEBUG] [app] Starting request: urn:request:2773b70c-ddaa-4bee-ad9e-19e3e16e2404 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:29,244 [237] [DEBUG] [app] Ending request: urn:request:a6af3835-6c05-4fc4-9423-3a2ff9c05b9a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6af3835-6c05-4fc4-9423-3a2ff9c05b9a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:04:29,245 [236] [DEBUG] [app] Ending request: urn:request:2773b70c-ddaa-4bee-ad9e-19e3e16e2404 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2773b70c-ddaa-4bee-ad9e-19e3e16e2404', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:04:29,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:04:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:29,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:04:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:04:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:29,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:29,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:29,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:29,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:29,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:29,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:29,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:29,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:29,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:29,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:29,257 [235] [DEBUG] [app] Ending request: urn:request:bea4f100-486f-4522-8948-dd5443a852af (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bea4f100-486f-4522-8948-dd5443a852af', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:29,257 [238] [DEBUG] [app] Ending request: urn:request:dc6ba880-40d3-4f35-8b21-f03b4d644543 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc6ba880-40d3-4f35-8b21-f03b4d644543', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:29,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:29,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:29,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:04:29,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.023)
storagereplication stdout | 2025-02-07 05:04:30,495 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:04:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:04:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:32.625945+00:00 (in 1.004351 seconds)
namespacegcworker stdout | 2025-02-07 05:04:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:31 UTC)" (scheduled at 2025-02-07 05:04:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:04:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:04:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 31, 621866), True, datetime.datetime(2025, 2, 7, 5, 4, 31, 621866), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:04:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:04:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:04:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:04:32,258 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:04:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:04:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:31.621131+00:00 (in 58.994703 seconds)
namespacegcworker stdout | 2025-02-07 05:04:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:32 UTC)" (scheduled at 2025-02-07 05:04:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:04:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:04:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:04:33,402 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:04:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:04:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:03.919286+00:00 (in 29.999515 seconds)
buildlogsarchiver stdout | 2025-02-07 05:04:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:03 UTC)" (scheduled at 2025-02-07 05:04:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:04:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 4, 33, 920055), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:04:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:04:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:04:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:56.713547+00:00 (in 18.997160 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:37 UTC)" (scheduled at 2025-02-07 05:04:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:04:37,742 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:04:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:04:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:48.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 05:04:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:48 UTC)" (scheduled at 2025-02-07 05:04:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:04:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:04:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 38, 732863), True, datetime.datetime(2025, 2, 7, 5, 4, 38, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:04:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:04:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:04:38,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:04:38,769 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:04:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:04:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:54.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 05:04:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:10 UTC)" (scheduled at 2025-02-07 05:04:40.424250+00:00)
gcworker stdout | 2025-02-07 05:04:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:04:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904380431, None, 1, 0])
gcworker stdout | 2025-02-07 05:04:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:04:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:04:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:04:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:11.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 05:04:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:11 UTC)" (scheduled at 2025-02-07 05:04:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:04:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901081836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:04:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:04:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:04:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:04:42,657 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:04:43,231 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:04:43,825 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:04:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:04:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:59.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 05:04:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:59 UTC)" (scheduled at 2025-02-07 05:04:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:04:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:04:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:04:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:04:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:04:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:04:44,236 [235] [DEBUG] [app] Starting request: urn:request:2b3ecf11-5a00-4e9e-88cd-b9d21de20f9d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:44,236 [238] [DEBUG] [app] Starting request: urn:request:8d39b295-9657-48f5-bbfa-c86c2e8baef7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:04:44,239 [246] [DEBUG] [app] Starting request: urn:request:fd337bad-b552-4b37-8da0-1867c8cbd19a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:44,239 [245] [DEBUG] [app] Starting request: urn:request:0cea4e5d-9540-49e0-b01e-cb9c0dc38c8c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:44,239 [246] [DEBUG] [app] Ending request: urn:request:fd337bad-b552-4b37-8da0-1867c8cbd19a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd337bad-b552-4b37-8da0-1867c8cbd19a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:04:44,239 [245] [DEBUG] [app] Ending request: urn:request:0cea4e5d-9540-49e0-b01e-cb9c0dc38c8c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0cea4e5d-9540-49e0-b01e-cb9c0dc38c8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:04:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:04:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:04:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:44,242 [235] [DEBUG] [app] Starting request: urn:request:c6484718-ffc8-41e8-8725-a34a59f4ffbd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:44,242 [235] [DEBUG] [app] Ending request: urn:request:c6484718-ffc8-41e8-8725-a34a59f4ffbd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c6484718-ffc8-41e8-8725-a34a59f4ffbd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:04:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:04:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:44,243 [237] [DEBUG] [app] Starting request: urn:request:1442407b-7a5c-4539-9dd2-334fb78cfeed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:44,243 [237] [DEBUG] [app] Ending request: urn:request:1442407b-7a5c-4539-9dd2-334fb78cfeed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1442407b-7a5c-4539-9dd2-334fb78cfeed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:04:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:04:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:44,253 [235] [DEBUG] [app] Ending request: urn:request:2b3ecf11-5a00-4e9e-88cd-b9d21de20f9d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b3ecf11-5a00-4e9e-88cd-b9d21de20f9d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:04:44,254 [238] [DEBUG] [app] Ending request: urn:request:8d39b295-9657-48f5-bbfa-c86c2e8baef7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d39b295-9657-48f5-bbfa-c86c2e8baef7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 05:04:45,140 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:04:46,043 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:04:46,106 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:04:47,289 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:04:47,337 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:04:47,346 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:04:47,351 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:04:47,355 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:04:47,359 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:47,695 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:04:48,132 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:04:48,136 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:04:48,366 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-secscan stdout | 2025-02-07 05:04:48,370 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:04:48,373 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:04:48,375 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:04:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:04:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:50.814677+00:00 (in 2.082041 seconds)
notificationworker stdout | 2025-02-07 05:04:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:58 UTC)" (scheduled at 2025-02-07 05:04:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:04:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:04:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 48, 732913), True, datetime.datetime(2025, 2, 7, 5, 4, 48, 732913), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:04:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:04:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:04:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:04:58 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:04:48,759 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:04:49,191 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:04:49,354 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:04:49,616 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:04:49,678 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:04:49,680 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:04:49,683 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:04:49,688 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:04:49,691 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:04:49,697 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:04:49,705 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:04:49,707 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:04:49,710 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 05:04:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:04:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:58.732147+00:00 (in 7.917018 seconds)
notificationworker stdout | 2025-02-07 05:04:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:50 UTC)" (scheduled at 2025-02-07 05:04:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:04:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:04:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:04:51,018 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:04:51,150 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:04:51,557 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
exportactionlogsworker stdout | 2025-02-07 05:04:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:04:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:04:59.937899+00:00 (in 6.999773 seconds)
exportactionlogsworker stdout | 2025-02-07 05:04:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:09:52 UTC)" (scheduled at 2025-02-07 05:04:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:04:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 4, 52, 938396), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:04:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 4, 52, 938396), True, datetime.datetime(2025, 2, 7, 5, 4, 52, 938396), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:04:52,946 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 4, 52, 938396), True, datetime.datetime(2025, 2, 7, 5, 4, 52, 938396), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 4, 52, 938396), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:04:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:04:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:09:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:04:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:04:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:10.424250+00:00 (in 16.000712 seconds)
gcworker stdout | 2025-02-07 05:04:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:24 UTC)" (scheduled at 2025-02-07 05:04:54.423055+00:00)
gcworker stdout | 2025-02-07 05:04:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:04:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:04:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:04:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:56.118855+00:00 (in 59.999500 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:04:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:56 UTC)" (scheduled at 2025-02-07 05:04:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:04:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:04:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:04:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:37.715878+00:00 (in 41.001871 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:56 UTC)" (scheduled at 2025-02-07 05:04:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:04:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:04:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:04:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:08.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 05:04:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:08 UTC)" (scheduled at 2025-02-07 05:04:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:04:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:04:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 4, 58, 732842), True, datetime.datetime(2025, 2, 7, 5, 4, 58, 732842), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:04:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:04:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:04:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:04:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:04:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:14.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 05:04:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:14 UTC)" (scheduled at 2025-02-07 05:04:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:04:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:04:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:04:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:04:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:04:59,235 [236] [DEBUG] [app] Starting request: urn:request:fb142da7-a5e7-4a0a-9d7a-7d87dda9e1da (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:59,235 [235] [DEBUG] [app] Starting request: urn:request:1720df7e-042d-437f-835e-a41237b07b29 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:04:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:04:59,238 [239] [DEBUG] [app] Starting request: urn:request:0a31ecd9-ecd5-49fe-ae01-d44a7bcb78b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:59,238 [246] [DEBUG] [app] Starting request: urn:request:f1b63c9b-2698-4738-9a60-2853e5cfd567 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:04:59,238 [246] [DEBUG] [app] Ending request: urn:request:f1b63c9b-2698-4738-9a60-2853e5cfd567 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f1b63c9b-2698-4738-9a60-2853e5cfd567', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:04:59,238 [239] [DEBUG] [app] Ending request: urn:request:0a31ecd9-ecd5-49fe-ae01-d44a7bcb78b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a31ecd9-ecd5-49fe-ae01-d44a7bcb78b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:04:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:04:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:04:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:04:59,242 [237] [DEBUG] [app] Starting request: urn:request:dd3e0b5b-1375-4d2d-981e-40ae855ac2dd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:59,242 [238] [DEBUG] [app] Starting request: urn:request:429246a8-b081-4c45-9000-779407ae581f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:04:59,243 [237] [DEBUG] [app] Ending request: urn:request:dd3e0b5b-1375-4d2d-981e-40ae855ac2dd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd3e0b5b-1375-4d2d-981e-40ae855ac2dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:04:59,243 [238] [DEBUG] [app] Ending request: urn:request:429246a8-b081-4c45-9000-779407ae581f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:429246a8-b081-4c45-9000-779407ae581f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:04:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:04:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:04:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:04:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:04:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:04:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:04:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:04:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:04:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:04:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:04:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:04:59,254 [236] [DEBUG] [app] Ending request: urn:request:fb142da7-a5e7-4a0a-9d7a-7d87dda9e1da (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fb142da7-a5e7-4a0a-9d7a-7d87dda9e1da', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:59,254 [235] [DEBUG] [app] Ending request: urn:request:1720df7e-042d-437f-835e-a41237b07b29 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1720df7e-042d-437f-835e-a41237b07b29', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:04:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:04:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:04:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:04:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:04:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 05:04:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:04:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:04.936688+00:00 (in 4.998319 seconds)
exportactionlogsworker stdout | 2025-02-07 05:04:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:59 UTC)" (scheduled at 2025-02-07 05:04:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:04:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:04:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:05:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:05:00,532 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:05:02,274 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:05:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:05:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:23.420886+00:00 (in 20.001337 seconds)
repositorygcworker stdout | 2025-02-07 05:05:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:03 UTC)" (scheduled at 2025-02-07 05:05:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:05:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:05:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 3, 419853), True, datetime.datetime(2025, 2, 7, 5, 5, 3, 419853), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:05:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:05:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:05:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:05:03,437 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:05:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:05:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:33.919286+00:00 (in 29.999461 seconds)
buildlogsarchiver stdout | 2025-02-07 05:05:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:33 UTC)" (scheduled at 2025-02-07 05:05:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:05:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 5, 3, 920078), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:05:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:05:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:05:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:05:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:05:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:59.937899+00:00 (in 55.000752 seconds)
exportactionlogsworker stdout | 2025-02-07 05:05:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:04 UTC)" (scheduled at 2025-02-07 05:05:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:05:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:05:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 4, 937491), True, datetime.datetime(2025, 2, 7, 5, 5, 4, 937491), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:05:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:05:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:05:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:05:07,771 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:05:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:05:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:18.732147+00:00 (in 9.999571 seconds)
notificationworker stdout | 2025-02-07 05:05:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:18 UTC)" (scheduled at 2025-02-07 05:05:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:05:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:05:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 8, 732845), True, datetime.datetime(2025, 2, 7, 5, 5, 8, 732845), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:05:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:05:08,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:05:08,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:05:08,789 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:05:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:05:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:24.423055+00:00 (in 13.998367 seconds)
gcworker stdout | 2025-02-07 05:05:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:40 UTC)" (scheduled at 2025-02-07 05:05:10.424250+00:00)
gcworker stdout | 2025-02-07 05:05:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:05:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904410431, None, 1, 0])
gcworker stdout | 2025-02-07 05:05:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:05:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:05:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:05:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:41.831521+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2025-02-07 05:05:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:41 UTC)" (scheduled at 2025-02-07 05:05:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:05:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901111836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:05:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:05:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:05:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:05:12,680 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:05:13,267 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:05:13,831 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:05:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:05:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:29.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 05:05:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:29 UTC)" (scheduled at 2025-02-07 05:05:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:05:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:05:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:05:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:05:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:05:14,235 [238] [DEBUG] [app] Starting request: urn:request:6c5a79de-3e7d-4d7c-81aa-cde496e3b3e8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:14,235 [235] [DEBUG] [app] Starting request: urn:request:7f192638-4a80-4f43-afac-2756e05cd972 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:05:14,238 [239] [DEBUG] [app] Starting request: urn:request:336d18bb-027f-4b66-9631-90e5cfa51919 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:05:14,238 [245] [DEBUG] [app] Starting request: urn:request:9687d56b-88d8-4c81-a4da-3c40d4831d9d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:05:14,239 [245] [DEBUG] [app] Ending request: urn:request:9687d56b-88d8-4c81-a4da-3c40d4831d9d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9687d56b-88d8-4c81-a4da-3c40d4831d9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:05:14,239 [239] [DEBUG] [app] Ending request: urn:request:336d18bb-027f-4b66-9631-90e5cfa51919 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:336d18bb-027f-4b66-9631-90e5cfa51919', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:05:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:05:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:14,241 [237] [DEBUG] [app] Starting request: urn:request:5387e411-a10f-4c8a-b780-873221ea0d27 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:14,241 [236] [DEBUG] [app] Starting request: urn:request:df955757-9ce1-4031-a9e2-cf1fec36dc5e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:14,242 [237] [DEBUG] [app] Ending request: urn:request:5387e411-a10f-4c8a-b780-873221ea0d27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5387e411-a10f-4c8a-b780-873221ea0d27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:05:14,242 [236] [DEBUG] [app] Ending request: urn:request:df955757-9ce1-4031-a9e2-cf1fec36dc5e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:df955757-9ce1-4031-a9e2-cf1fec36dc5e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:05:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:14,253 [238] [DEBUG] [app] Ending request: urn:request:6c5a79de-3e7d-4d7c-81aa-cde496e3b3e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c5a79de-3e7d-4d7c-81aa-cde496e3b3e8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:05:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:05:14,254 [235] [DEBUG] [app] Ending request: urn:request:7f192638-4a80-4f43-afac-2756e05cd972 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f192638-4a80-4f43-afac-2756e05cd972', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:05:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:05:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:05:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
expiredappspecifictokenworker stdout | 2025-02-07 05:05:15,178 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:05:16,079 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:05:16,142 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
autopruneworker stdout | 2025-02-07 05:05:17,318 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:05:17,344 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:05:17,353 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:05:17,359 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:05:17,362 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:05:17,365 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:17,728 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:05:18,141 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:05:18,173 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:05:18,379 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:05:18,383 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:05:18,386 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:05:18,401 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:05:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:05:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:28.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 05:05:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:28 UTC)" (scheduled at 2025-02-07 05:05:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:05:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:05:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 18, 732877), True, datetime.datetime(2025, 2, 7, 5, 5, 18, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:05:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:05:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:05:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:05:18,775 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:05:19,207 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:05:19,368 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:05:19,629 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:05:19,685 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:05:19,690 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:05:19,693 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:05:19,698 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:05:19,700 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:05:19,706 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:05:19,713 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:05:19,718 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:05:19,721 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
buildlogsarchiver stdout | 2025-02-07 05:05:21,054 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:05:21,186 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:05:21,594 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:05:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:05:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:03.419110+00:00 (in 39.997759 seconds)
repositorygcworker stdout | 2025-02-07 05:05:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:23 UTC)" (scheduled at 2025-02-07 05:05:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:05:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:05:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:05:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:05:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:40.424250+00:00 (in 16.000756 seconds)
gcworker stdout | 2025-02-07 05:05:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:54 UTC)" (scheduled at 2025-02-07 05:05:24.423055+00:00)
gcworker stdout | 2025-02-07 05:05:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:05:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:05:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:05:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:05:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:38.732147+00:00 (in 9.999572 seconds)
notificationworker stdout | 2025-02-07 05:05:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:38 UTC)" (scheduled at 2025-02-07 05:05:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:05:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:05:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 28, 732848), True, datetime.datetime(2025, 2, 7, 5, 5, 28, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:05:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:05:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:05:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:05:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:05:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:44.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 05:05:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:44 UTC)" (scheduled at 2025-02-07 05:05:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:05:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:05:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:05:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:05:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:05:29,235 [237] [DEBUG] [app] Starting request: urn:request:864fc719-24d6-4939-bd37-62d9fe3b6979 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:29,235 [235] [DEBUG] [app] Starting request: urn:request:1eb726ef-d73a-45fe-9085-c8b42cadc0ab (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:05:29,238 [245] [DEBUG] [app] Starting request: urn:request:dc9e279c-d43e-4491-bb2a-e79744b13ef8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:05:29,238 [239] [DEBUG] [app] Starting request: urn:request:789a4dac-aad1-498d-a784-0a6e27c1590c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:05:29,239 [245] [DEBUG] [app] Ending request: urn:request:dc9e279c-d43e-4491-bb2a-e79744b13ef8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc9e279c-d43e-4491-bb2a-e79744b13ef8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:05:29,239 [239] [DEBUG] [app] Ending request: urn:request:789a4dac-aad1-498d-a784-0a6e27c1590c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:789a4dac-aad1-498d-a784-0a6e27c1590c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:05:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:05:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:05:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:29,241 [238] [DEBUG] [app] Starting request: urn:request:f23e082d-a32d-45af-a8c5-6ba10d91f29b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:29,242 [236] [DEBUG] [app] Starting request: urn:request:ecfcbe6d-d2b2-436a-aa1a-107e37c59e69 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:29,242 [238] [DEBUG] [app] Ending request: urn:request:f23e082d-a32d-45af-a8c5-6ba10d91f29b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f23e082d-a32d-45af-a8c5-6ba10d91f29b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:05:29,242 [236] [DEBUG] [app] Ending request: urn:request:ecfcbe6d-d2b2-436a-aa1a-107e37c59e69 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ecfcbe6d-d2b2-436a-aa1a-107e37c59e69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:05:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:29,253 [237] [DEBUG] [app] Ending request: urn:request:864fc719-24d6-4939-bd37-62d9fe3b6979 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:864fc719-24d6-4939-bd37-62d9fe3b6979', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:05:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:05:29,254 [235] [DEBUG] [app] Ending request: urn:request:1eb726ef-d73a-45fe-9085-c8b42cadc0ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1eb726ef-d73a-45fe-9085-c8b42cadc0ab', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:05:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:05:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 05:05:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
storagereplication stdout | 2025-02-07 05:05:30,569 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:05:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:05:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:32.625945+00:00 (in 1.004359 seconds)
namespacegcworker stdout | 2025-02-07 05:05:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:31 UTC)" (scheduled at 2025-02-07 05:05:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:05:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:05:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 31, 621874), True, datetime.datetime(2025, 2, 7, 5, 5, 31, 621874), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:05:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:05:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:05:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:05:32,306 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:05:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:05:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:31.621131+00:00 (in 58.994727 seconds)
namespacegcworker stdout | 2025-02-07 05:05:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:32 UTC)" (scheduled at 2025-02-07 05:05:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:05:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:05:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:05:33,446 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:05:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:05:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:03.919286+00:00 (in 29.999554 seconds)
buildlogsarchiver stdout | 2025-02-07 05:05:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:03 UTC)" (scheduled at 2025-02-07 05:05:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:05:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 5, 33, 919978), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:05:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:05:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:05:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:56.713547+00:00 (in 18.997135 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:37 UTC)" (scheduled at 2025-02-07 05:05:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:05:37,807 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:05:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:05:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:48.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 05:05:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:48 UTC)" (scheduled at 2025-02-07 05:05:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:05:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:05:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 38, 732863), True, datetime.datetime(2025, 2, 7, 5, 5, 38, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:05:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:05:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:05:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:05:38,818 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:05:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:05:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:54.423055+00:00 (in 13.998334 seconds)
gcworker stdout | 2025-02-07 05:05:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:10 UTC)" (scheduled at 2025-02-07 05:05:40.424250+00:00)
gcworker stdout | 2025-02-07 05:05:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:05:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904440431, None, 1, 0])
gcworker stdout | 2025-02-07 05:05:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:05:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:05:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:05:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:11.831521+00:00 (in 29.999584 seconds)
autopruneworker stdout | 2025-02-07 05:05:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:11 UTC)" (scheduled at 2025-02-07 05:05:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:05:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901141835, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:05:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:05:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:05:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:05:42,716 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:05:43,296 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:05:43,838 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:05:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:05:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:59.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 05:05:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:59 UTC)" (scheduled at 2025-02-07 05:05:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:05:44,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:05:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:05:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:05:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:05:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:05:44,235 [238] [DEBUG] [app] Starting request: urn:request:5c9aa5d7-6cb3-40b3-8103-b15d10f8b9e5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:44,235 [235] [DEBUG] [app] Starting request: urn:request:e29708f3-6da2-4676-8201-2f625c964585 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:05:44,238 [239] [DEBUG] [app] Starting request: urn:request:f8b9af7a-bbc6-4bdd-a925-de5422233619 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:05:44,238 [246] [DEBUG] [app] Starting request: urn:request:ab88ff79-a011-4a90-9236-7a5bfccf5716 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:05:44,238 [239] [DEBUG] [app] Ending request: urn:request:f8b9af7a-bbc6-4bdd-a925-de5422233619 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f8b9af7a-bbc6-4bdd-a925-de5422233619', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:05:44,238 [246] [DEBUG] [app] Ending request: urn:request:ab88ff79-a011-4a90-9236-7a5bfccf5716 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab88ff79-a011-4a90-9236-7a5bfccf5716', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:05:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:05:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:44,242 [237] [DEBUG] [app] Starting request: urn:request:54ca9aa9-70da-4be5-82d0-7709325db467 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:44,242 [237] [DEBUG] [app] Ending request: urn:request:54ca9aa9-70da-4be5-82d0-7709325db467 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:54ca9aa9-70da-4be5-82d0-7709325db467', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:05:44,242 [236] [DEBUG] [app] Starting request: urn:request:39b208fb-d74e-401d-beda-8c7a2b7fac1e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:05:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:44,242 [236] [DEBUG] [app] Ending request: urn:request:39b208fb-d74e-401d-beda-8c7a2b7fac1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:39b208fb-d74e-401d-beda-8c7a2b7fac1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:05:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:05:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:44,253 [238] [DEBUG] [app] Ending request: urn:request:5c9aa5d7-6cb3-40b3-8103-b15d10f8b9e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5c9aa5d7-6cb3-40b3-8103-b15d10f8b9e5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:05:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:05:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:05:44,254 [235] [DEBUG] [app] Ending request: urn:request:e29708f3-6da2-4676-8201-2f625c964585 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e29708f3-6da2-4676-8201-2f625c964585', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:05:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:05:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 05:05:45,214 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:05:46,115 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:05:46,152 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:05:47,353 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
autopruneworker stdout | 2025-02-07 05:05:47,356 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-web stdout | 2025-02-07 05:05:47,359 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:05:47,365 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:05:47,368 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:05:47,372 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:47,765 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:05:48,177 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:05:48,209 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:05:48,387 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:05:48,391 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:05:48,393 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:05:48,438 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:05:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:05:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:50.814677+00:00 (in 2.082082 seconds)
notificationworker stdout | 2025-02-07 05:05:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:58 UTC)" (scheduled at 2025-02-07 05:05:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:05:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:05:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 48, 732919), True, datetime.datetime(2025, 2, 7, 5, 5, 48, 732919), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:05:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:05:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:05:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:05:58 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:05:48,811 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:05:49,224 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:05:49,402 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:05:49,641 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:05:49,694 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:05:49,701 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:05:49,704 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:05:49,706 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:05:49,709 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:05:49,716 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:05:49,724 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:05:49,727 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:05:49,730 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 05:05:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:05:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:05:58.732147+00:00 (in 7.917002 seconds)
notificationworker stdout | 2025-02-07 05:05:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:50 UTC)" (scheduled at 2025-02-07 05:05:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:05:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:05:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:05:51,090 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:05:51,222 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:05:51,630 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:05:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:05:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:10.424250+00:00 (in 16.000704 seconds)
gcworker stdout | 2025-02-07 05:05:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:24 UTC)" (scheduled at 2025-02-07 05:05:54.423055+00:00)
gcworker stdout | 2025-02-07 05:05:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:05:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:05:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:05:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:56.118855+00:00 (in 59.999499 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:05:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:56 UTC)" (scheduled at 2025-02-07 05:05:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:05:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:05:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:05:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:37.715878+00:00 (in 41.001843 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:56 UTC)" (scheduled at 2025-02-07 05:05:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:05:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:05:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:05:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:08.732147+00:00 (in 9.999483 seconds)
notificationworker stdout | 2025-02-07 05:05:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:08 UTC)" (scheduled at 2025-02-07 05:05:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:05:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:05:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 5, 58, 732971), True, datetime.datetime(2025, 2, 7, 5, 5, 58, 732971), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:05:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:05:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:05:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:05:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:05:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:14.225378+00:00 (in 14.999576 seconds)
quotatotalworker stdout | 2025-02-07 05:05:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:14 UTC)" (scheduled at 2025-02-07 05:05:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:05:59,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:05:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:05:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:05:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:05:59,235 [235] [DEBUG] [app] Starting request: urn:request:81dd6a3c-f7c5-4287-adc4-fd84b96e14f4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:59,235 [238] [DEBUG] [app] Starting request: urn:request:b492b0b9-a92f-4624-ab76-2b518a5bfca5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:05:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:05:59,238 [239] [DEBUG] [app] Starting request: urn:request:9cedf3e7-2e0c-4b8a-b1c4-954b04280fa6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:05:59,238 [239] [DEBUG] [app] Ending request: urn:request:9cedf3e7-2e0c-4b8a-b1c4-954b04280fa6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9cedf3e7-2e0c-4b8a-b1c4-954b04280fa6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:05:59,238 [244] [DEBUG] [app] Starting request: urn:request:9255d7fe-1249-4ed1-bd5f-d1c50e5df27c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:05:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:05:59,239 [244] [DEBUG] [app] Ending request: urn:request:9255d7fe-1249-4ed1-bd5f-d1c50e5df27c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9255d7fe-1249-4ed1-bd5f-d1c50e5df27c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:05:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:05:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:05:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:05:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:59,241 [238] [DEBUG] [app] Starting request: urn:request:79221993-757a-48a0-adf8-3148f32b9d16 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:05:59,241 [238] [DEBUG] [app] Ending request: urn:request:79221993-757a-48a0-adf8-3148f32b9d16 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79221993-757a-48a0-adf8-3148f32b9d16', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:05:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:05:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:59,242 [237] [DEBUG] [app] Starting request: urn:request:23d33fe9-fd4a-42c2-81be-5c89d7dfc5cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:05:59,243 [237] [DEBUG] [app] Ending request: urn:request:23d33fe9-fd4a-42c2-81be-5c89d7dfc5cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:23d33fe9-fd4a-42c2-81be-5c89d7dfc5cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:05:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:05:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:05:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:05:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:05:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:05:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:05:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:05:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:05:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:05:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:05:59,253 [238] [DEBUG] [app] Ending request: urn:request:b492b0b9-a92f-4624-ab76-2b518a5bfca5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b492b0b9-a92f-4624-ab76-2b518a5bfca5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:05:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:05:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:05:59,254 [235] [DEBUG] [app] Ending request: urn:request:81dd6a3c-f7c5-4287-adc4-fd84b96e14f4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81dd6a3c-f7c5-4287-adc4-fd84b96e14f4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:05:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:05:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:05:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:05:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
exportactionlogsworker stdout | 2025-02-07 05:05:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:05:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:04.936688+00:00 (in 4.998258 seconds)
exportactionlogsworker stdout | 2025-02-07 05:05:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:59 UTC)" (scheduled at 2025-02-07 05:05:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:05:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:05:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:06:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:06:00,594 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:06:02,342 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:06:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:06:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:23.420886+00:00 (in 20.001355 seconds)
repositorygcworker stdout | 2025-02-07 05:06:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:03 UTC)" (scheduled at 2025-02-07 05:06:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:06:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:06:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 3, 419797), True, datetime.datetime(2025, 2, 7, 5, 6, 3, 419797), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:06:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:06:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:06:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:06:03,482 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:06:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:06:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:33.919286+00:00 (in 29.999569 seconds)
buildlogsarchiver stdout | 2025-02-07 05:06:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:33 UTC)" (scheduled at 2025-02-07 05:06:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:06:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 6, 3, 919956), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:06:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:06:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:06:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:06:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:06:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:59.937899+00:00 (in 55.000720 seconds)
exportactionlogsworker stdout | 2025-02-07 05:06:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:04 UTC)" (scheduled at 2025-02-07 05:06:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:06:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:06:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 4, 937530), True, datetime.datetime(2025, 2, 7, 5, 6, 4, 937530), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:06:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:06:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:06:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:06:07,821 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:06:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:06:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:18.732147+00:00 (in 9.999501 seconds)
notificationworker stdout | 2025-02-07 05:06:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:18 UTC)" (scheduled at 2025-02-07 05:06:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:06:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:06:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 8, 732959), True, datetime.datetime(2025, 2, 7, 5, 6, 8, 732959), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:06:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:06:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:06:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:06:08,828 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:06:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:06:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:24.423055+00:00 (in 13.998394 seconds)
gcworker stdout | 2025-02-07 05:06:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:40 UTC)" (scheduled at 2025-02-07 05:06:10.424250+00:00)
gcworker stdout | 2025-02-07 05:06:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:06:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904470431, None, 1, 0])
gcworker stdout | 2025-02-07 05:06:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:06:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:06:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:06:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:41.831521+00:00 (in 29.999569 seconds)
autopruneworker stdout | 2025-02-07 05:06:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:41 UTC)" (scheduled at 2025-02-07 05:06:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:06:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901171836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:06:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:06:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:06:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:06:12,740 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:06:13,306 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:06:13,871 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:06:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:06:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:29.225378+00:00 (in 14.999539 seconds)
quotatotalworker stdout | 2025-02-07 05:06:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:29 UTC)" (scheduled at 2025-02-07 05:06:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:06:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:06:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:06:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:06:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:06:14,236 [237] [DEBUG] [app] Starting request: urn:request:469877b0-b908-4ebc-b18f-8b9bcfe72aa1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:14,236 [235] [DEBUG] [app] Starting request: urn:request:239ddfac-2732-4c2a-bf20-29ac581cc0cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:06:14,239 [245] [DEBUG] [app] Starting request: urn:request:ad769479-7373-487d-bd34-43cefa2257fe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:14,239 [246] [DEBUG] [app] Starting request: urn:request:85cef3ba-953b-420d-a259-cdc4ef26b7f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:14,239 [246] [DEBUG] [app] Ending request: urn:request:85cef3ba-953b-420d-a259-cdc4ef26b7f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85cef3ba-953b-420d-a259-cdc4ef26b7f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:14,239 [245] [DEBUG] [app] Ending request: urn:request:ad769479-7373-487d-bd34-43cefa2257fe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ad769479-7373-487d-bd34-43cefa2257fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:06:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:06:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:14,242 [235] [DEBUG] [app] Starting request: urn:request:6c6410f3-f906-4b71-aacd-357f88e9a40c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:14,242 [238] [DEBUG] [app] Starting request: urn:request:a4b268da-523d-48a1-8380-27c282878d2a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:14,242 [235] [DEBUG] [app] Ending request: urn:request:6c6410f3-f906-4b71-aacd-357f88e9a40c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6c6410f3-f906-4b71-aacd-357f88e9a40c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:06:14,243 [238] [DEBUG] [app] Ending request: urn:request:a4b268da-523d-48a1-8380-27c282878d2a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4b268da-523d-48a1-8380-27c282878d2a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:06:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:06:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:06:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:06:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:06:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:14,255 [235] [DEBUG] [app] Ending request: urn:request:239ddfac-2732-4c2a-bf20-29ac581cc0cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:239ddfac-2732-4c2a-bf20-29ac581cc0cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:14,255 [237] [DEBUG] [app] Ending request: urn:request:469877b0-b908-4ebc-b18f-8b9bcfe72aa1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:469877b0-b908-4ebc-b18f-8b9bcfe72aa1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:06:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:06:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:06:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 05:06:15,250 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:06:16,128 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:06:16,170 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:06:17,361 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:06:17,367 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:06:17,371 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:06:17,375 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:06:17,379 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:06:17,390 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:17,797 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:06:18,213 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:06:18,245 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:06:18,395 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:06:18,399 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:06:18,401 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:06:18,471 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:06:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:06:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:28.732147+00:00 (in 9.999530 seconds)
notificationworker stdout | 2025-02-07 05:06:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:28 UTC)" (scheduled at 2025-02-07 05:06:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:06:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:06:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 18, 732896), True, datetime.datetime(2025, 2, 7, 5, 6, 18, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:06:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:06:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:06:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:06:18,847 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:06:19,254 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:06:19,423 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:06:19,677 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:06:19,706 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:06:19,711 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:06:19,714 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:06:19,717 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:06:19,719 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:06:19,725 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:06:19,733 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:06:19,736 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:06:19,739 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
buildlogsarchiver stdout | 2025-02-07 05:06:21,115 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:06:21,258 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:06:21,660 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:06:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:06:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:03.419110+00:00 (in 39.997759 seconds)
repositorygcworker stdout | 2025-02-07 05:06:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:23 UTC)" (scheduled at 2025-02-07 05:06:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:06:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:06:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:06:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:06:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:40.424250+00:00 (in 16.000736 seconds)
gcworker stdout | 2025-02-07 05:06:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:54 UTC)" (scheduled at 2025-02-07 05:06:24.423055+00:00)
gcworker stdout | 2025-02-07 05:06:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:06:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:06:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:06:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:06:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:38.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 05:06:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:38 UTC)" (scheduled at 2025-02-07 05:06:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:06:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:06:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 28, 732890), True, datetime.datetime(2025, 2, 7, 5, 6, 28, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:06:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:06:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:06:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:06:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:06:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:44.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 05:06:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:44 UTC)" (scheduled at 2025-02-07 05:06:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:06:29,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:06:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:06:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:06:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:06:29,235 [238] [DEBUG] [app] Starting request: urn:request:c2cfaa6d-7cc9-4b88-b444-3bb5fa448d8c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:29,235 [235] [DEBUG] [app] Starting request: urn:request:37b55782-9ef1-4162-b4a7-afbb594d06a5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:06:29,238 [246] [DEBUG] [app] Starting request: urn:request:29c1e507-07b9-4e2f-be0f-82b5ec8f3727 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:29,238 [245] [DEBUG] [app] Starting request: urn:request:0fe773e0-1e8a-4a78-b19a-5496680807be (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:29,238 [245] [DEBUG] [app] Ending request: urn:request:0fe773e0-1e8a-4a78-b19a-5496680807be (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0fe773e0-1e8a-4a78-b19a-5496680807be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:29,238 [246] [DEBUG] [app] Ending request: urn:request:29c1e507-07b9-4e2f-be0f-82b5ec8f3727 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:29c1e507-07b9-4e2f-be0f-82b5ec8f3727', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:06:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:06:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:29,241 [238] [DEBUG] [app] Starting request: urn:request:02094670-603b-4f35-a55e-8153d64b229c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:29,241 [238] [DEBUG] [app] Ending request: urn:request:02094670-603b-4f35-a55e-8153d64b229c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02094670-603b-4f35-a55e-8153d64b229c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:06:29,241 [237] [DEBUG] [app] Starting request: urn:request:505898f8-1488-4ae5-9ad4-69c4f39df8ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:29,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:06:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:29,242 [237] [DEBUG] [app] Ending request: urn:request:505898f8-1488-4ae5-9ad4-69c4f39df8ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:505898f8-1488-4ae5-9ad4-69c4f39df8ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:06:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:06:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:06:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:06:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:29,253 [238] [DEBUG] [app] Ending request: urn:request:c2cfaa6d-7cc9-4b88-b444-3bb5fa448d8c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2cfaa6d-7cc9-4b88-b444-3bb5fa448d8c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:29,253 [235] [DEBUG] [app] Ending request: urn:request:37b55782-9ef1-4162-b4a7-afbb594d06a5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:37b55782-9ef1-4162-b4a7-afbb594d06a5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:06:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:06:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:06:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
storagereplication stdout | 2025-02-07 05:06:30,618 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:06:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:06:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:32.625945+00:00 (in 1.004352 seconds)
namespacegcworker stdout | 2025-02-07 05:06:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:31 UTC)" (scheduled at 2025-02-07 05:06:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:06:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:06:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 31, 621903), True, datetime.datetime(2025, 2, 7, 5, 6, 31, 621903), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:06:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:06:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:06:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:06:32,368 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:06:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:06:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:13.624301+00:00 (in 40.997903 seconds)
namespacegcworker stdout | 2025-02-07 05:06:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:32 UTC)" (scheduled at 2025-02-07 05:06:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:06:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:06:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:06:33,518 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:06:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:06:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:03.919286+00:00 (in 29.999562 seconds)
buildlogsarchiver stdout | 2025-02-07 05:06:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:03 UTC)" (scheduled at 2025-02-07 05:06:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:06:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 6, 33, 919981), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:06:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:06:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:06:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:56.713547+00:00 (in 18.997190 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:37 UTC)" (scheduled at 2025-02-07 05:06:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:06:37,858 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:06:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:06:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:48.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 05:06:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:48 UTC)" (scheduled at 2025-02-07 05:06:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:06:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:06:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 38, 732886), True, datetime.datetime(2025, 2, 7, 5, 6, 38, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:06:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:06:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:06:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:06:38,843 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:06:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:06:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:54.423055+00:00 (in 13.998378 seconds)
gcworker stdout | 2025-02-07 05:06:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:10 UTC)" (scheduled at 2025-02-07 05:06:40.424250+00:00)
gcworker stdout | 2025-02-07 05:06:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:06:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904500431, None, 1, 0])
gcworker stdout | 2025-02-07 05:06:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:06:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:06:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:06:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:11.831521+00:00 (in 29.999570 seconds)
autopruneworker stdout | 2025-02-07 05:06:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:11 UTC)" (scheduled at 2025-02-07 05:06:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:06:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901201836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:06:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:06:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:06:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:06:42,774 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:06:43,342 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:06:43,906 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:06:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:06:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:59.225378+00:00 (in 14.999533 seconds)
quotatotalworker stdout | 2025-02-07 05:06:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:59 UTC)" (scheduled at 2025-02-07 05:06:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:06:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:06:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:06:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:06:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:06:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:06:44,235 [237] [DEBUG] [app] Starting request: urn:request:4d5082e3-a944-4ca0-b493-01b869ecf54e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:44,235 [238] [DEBUG] [app] Starting request: urn:request:7d3c1367-5c3f-4775-8976-8fbe5a71880a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:06:44,238 [244] [DEBUG] [app] Starting request: urn:request:a4a6b2ff-a024-4d6c-8eb4-50d24dab8248 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:44,238 [246] [DEBUG] [app] Starting request: urn:request:d2e777d2-b21a-4078-bec6-33e6d81046de (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:44,239 [244] [DEBUG] [app] Ending request: urn:request:a4a6b2ff-a024-4d6c-8eb4-50d24dab8248 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4a6b2ff-a024-4d6c-8eb4-50d24dab8248', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:44,239 [246] [DEBUG] [app] Ending request: urn:request:d2e777d2-b21a-4078-bec6-33e6d81046de (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2e777d2-b21a-4078-bec6-33e6d81046de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:06:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:06:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:44,242 [237] [DEBUG] [app] Starting request: urn:request:181ef2fa-b24b-4289-bfde-95f4297b9d8a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:44,242 [236] [DEBUG] [app] Starting request: urn:request:cdae7816-647f-4012-a0c0-9ac441e05ba1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:44,242 [237] [DEBUG] [app] Ending request: urn:request:181ef2fa-b24b-4289-bfde-95f4297b9d8a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:181ef2fa-b24b-4289-bfde-95f4297b9d8a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:06:44,242 [236] [DEBUG] [app] Ending request: urn:request:cdae7816-647f-4012-a0c0-9ac441e05ba1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cdae7816-647f-4012-a0c0-9ac441e05ba1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:06:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:06:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:06:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:06:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:06:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:06:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:44,254 [238] [DEBUG] [app] Ending request: urn:request:7d3c1367-5c3f-4775-8976-8fbe5a71880a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d3c1367-5c3f-4775-8976-8fbe5a71880a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:44,254 [237] [DEBUG] [app] Ending request: urn:request:4d5082e3-a944-4ca0-b493-01b869ecf54e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4d5082e3-a944-4ca0-b493-01b869ecf54e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:06:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:06:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:06:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 05:06:45,286 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:06:46,164 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:06:46,207 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:06:47,370 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:06:47,374 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:06:47,378 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:06:47,382 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:06:47,387 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:06:47,427 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:47,834 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:06:48,231 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:06:48,274 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:06:48,403 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:06:48,406 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:06:48,409 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:06:48,507 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:06:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:06:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:50.814677+00:00 (in 2.082060 seconds)
notificationworker stdout | 2025-02-07 05:06:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:58 UTC)" (scheduled at 2025-02-07 05:06:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:06:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:06:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 48, 732891), True, datetime.datetime(2025, 2, 7, 5, 6, 48, 732891), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:06:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:06:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:06:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:06:58 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:06:48,884 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:06:49,290 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:06:49,447 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:06:49,711 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:06:49,717 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:06:49,720 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:06:49,723 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:06:49,727 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:06:49,730 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:06:49,733 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:06:49,740 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:06:49,743 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:06:49,747 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
notificationworker stdout | 2025-02-07 05:06:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:06:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:06:58.732147+00:00 (in 7.917023 seconds)
notificationworker stdout | 2025-02-07 05:06:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:50 UTC)" (scheduled at 2025-02-07 05:06:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:06:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:06:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:06:51,130 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:06:51,294 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:06:51,696 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:06:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:06:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:10.424250+00:00 (in 16.000757 seconds)
gcworker stdout | 2025-02-07 05:06:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:24 UTC)" (scheduled at 2025-02-07 05:06:54.423055+00:00)
gcworker stdout | 2025-02-07 05:06:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:06:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:06:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:06:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:56.118855+00:00 (in 59.999513 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:06:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:56 UTC)" (scheduled at 2025-02-07 05:06:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:06:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:06:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:06:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:37.715878+00:00 (in 41.001850 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:56 UTC)" (scheduled at 2025-02-07 05:06:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:06:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:06:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:06:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:08.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 05:06:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:08 UTC)" (scheduled at 2025-02-07 05:06:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:06:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:06:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 6, 58, 732848), True, datetime.datetime(2025, 2, 7, 5, 6, 58, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:06:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:06:58,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:06:58,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:06:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:06:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:14.225378+00:00 (in 14.999543 seconds)
quotatotalworker stdout | 2025-02-07 05:06:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:14 UTC)" (scheduled at 2025-02-07 05:06:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:06:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:06:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:06:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:06:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:06:59,235 [237] [DEBUG] [app] Starting request: urn:request:a67da19e-5e10-4601-9cc5-646f8341a5c9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:59,235 [235] [DEBUG] [app] Starting request: urn:request:329a8986-6713-4285-a211-d73cf979ac15 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:06:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:06:59,238 [239] [DEBUG] [app] Starting request: urn:request:a14dd00c-9c45-46f8-88de-3ff23fc6c7c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:59,238 [246] [DEBUG] [app] Starting request: urn:request:9cda62da-f914-4823-a196-4a7bcd7d1648 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:06:59,238 [239] [DEBUG] [app] Ending request: urn:request:a14dd00c-9c45-46f8-88de-3ff23fc6c7c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a14dd00c-9c45-46f8-88de-3ff23fc6c7c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:59,238 [246] [DEBUG] [app] Ending request: urn:request:9cda62da-f914-4823-a196-4a7bcd7d1648 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9cda62da-f914-4823-a196-4a7bcd7d1648', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:06:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:06:59,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:06:59,238 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:59,238 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:06:59,241 [235] [DEBUG] [app] Starting request: urn:request:4632900e-b93f-46a3-ba34-9c384a3eaf1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:59,241 [238] [DEBUG] [app] Starting request: urn:request:2567491b-4533-474f-b526-c66f6741c96a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:06:59,241 [235] [DEBUG] [app] Ending request: urn:request:4632900e-b93f-46a3-ba34-9c384a3eaf1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4632900e-b93f-46a3-ba34-9c384a3eaf1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:06:59,241 [238] [DEBUG] [app] Ending request: urn:request:2567491b-4533-474f-b526-c66f6741c96a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2567491b-4533-474f-b526-c66f6741c96a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:06:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:06:59,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:06:59,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:06:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:06:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:06:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:06:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:06:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:06:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:06:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:06:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:06:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:59,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:06:59,253 [237] [DEBUG] [app] Ending request: urn:request:a67da19e-5e10-4601-9cc5-646f8341a5c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a67da19e-5e10-4601-9cc5-646f8341a5c9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:59,253 [235] [DEBUG] [app] Ending request: urn:request:329a8986-6713-4285-a211-d73cf979ac15 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:329a8986-6713-4285-a211-d73cf979ac15', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:06:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:06:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:06:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:06:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:06:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:06:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:06:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:06:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:04.936688+00:00 (in 4.998235 seconds)
exportactionlogsworker stdout | 2025-02-07 05:06:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:59 UTC)" (scheduled at 2025-02-07 05:06:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:06:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:06:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:07:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:07:00,654 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:07:02,394 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:07:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:07:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:23.420886+00:00 (in 20.001309 seconds)
repositorygcworker stdout | 2025-02-07 05:07:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:03 UTC)" (scheduled at 2025-02-07 05:07:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:07:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:07:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 3, 419911), True, datetime.datetime(2025, 2, 7, 5, 7, 3, 419911), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:07:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:07:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:07:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:07:03,555 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:07:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:07:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:33.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 05:07:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:33 UTC)" (scheduled at 2025-02-07 05:07:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:07:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 7, 3, 919999), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:07:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:07:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:07:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:07:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:07:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:59.937899+00:00 (in 55.000773 seconds)
exportactionlogsworker stdout | 2025-02-07 05:07:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:04 UTC)" (scheduled at 2025-02-07 05:07:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:07:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:07:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 4, 937362), True, datetime.datetime(2025, 2, 7, 5, 7, 4, 937362), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:07:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:07:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:07:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:07:07,881 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:07:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:18.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 05:07:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:18 UTC)" (scheduled at 2025-02-07 05:07:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:07:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:07:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 8, 732923), True, datetime.datetime(2025, 2, 7, 5, 7, 8, 732923), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:07:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:07:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:07:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:07:08,879 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:07:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:07:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:24.423055+00:00 (in 13.998403 seconds)
gcworker stdout | 2025-02-07 05:07:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:40 UTC)" (scheduled at 2025-02-07 05:07:10.424250+00:00)
gcworker stdout | 2025-02-07 05:07:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:07:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904530431, None, 1, 0])
gcworker stdout | 2025-02-07 05:07:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:07:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:07:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:07:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:41.831521+00:00 (in 29.999531 seconds)
autopruneworker stdout | 2025-02-07 05:07:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:41 UTC)" (scheduled at 2025-02-07 05:07:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:07:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901231836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:07:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:07:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:07:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:07:12,810 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:07:13,378 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
namespacegcworker stdout | 2025-02-07 05:07:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:07:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:31.621131+00:00 (in 17.996391 seconds)
namespacegcworker stdout | 2025-02-07 05:07:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:12:13 UTC)" (scheduled at 2025-02-07 05:07:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:07:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 7, 13, 624985), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:07:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 7, 13, 624985), True, datetime.datetime(2025, 2, 7, 5, 7, 13, 624985), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:07:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 7, 13, 624985), True, datetime.datetime(2025, 2, 7, 5, 7, 13, 624985), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 7, 13, 624985), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:07:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:07:13,634 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:12:13 UTC)" executed successfully
logrotateworker stdout | 2025-02-07 05:07:13,942 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:07:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:07:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:29.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 05:07:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:29 UTC)" (scheduled at 2025-02-07 05:07:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:07:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:07:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:07:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:07:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:07:14,235 [237] [DEBUG] [app] Starting request: urn:request:3394b6c0-6901-4688-9264-d92f9ab98c7d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:14,235 [238] [DEBUG] [app] Starting request: urn:request:38637162-7909-4f8d-a7fa-ed975297b77e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:07:14,238 [239] [DEBUG] [app] Starting request: urn:request:0f11705a-e213-4cb9-8cdf-e7f990e32973 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:14,238 [246] [DEBUG] [app] Starting request: urn:request:e3fb150c-e595-44e2-b8e3-cf1e278491e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:14,238 [239] [DEBUG] [app] Ending request: urn:request:0f11705a-e213-4cb9-8cdf-e7f990e32973 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f11705a-e213-4cb9-8cdf-e7f990e32973', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:07:14,239 [246] [DEBUG] [app] Ending request: urn:request:e3fb150c-e595-44e2-b8e3-cf1e278491e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e3fb150c-e595-44e2-b8e3-cf1e278491e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:07:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:07:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:07:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:14,241 [238] [DEBUG] [app] Starting request: urn:request:79941991-81eb-429e-8e17-eaf8941f52ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:14,241 [238] [DEBUG] [app] Ending request: urn:request:79941991-81eb-429e-8e17-eaf8941f52ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79941991-81eb-429e-8e17-eaf8941f52ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:07:14,241 [235] [DEBUG] [app] Starting request: urn:request:1aadc72b-fd3d-457a-bbf1-ef98737dfbf0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:14,242 [235] [DEBUG] [app] Ending request: urn:request:1aadc72b-fd3d-457a-bbf1-ef98737dfbf0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1aadc72b-fd3d-457a-bbf1-ef98737dfbf0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:07:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:07:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:14,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:14,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:14,252 [238] [DEBUG] [app] Ending request: urn:request:38637162-7909-4f8d-a7fa-ed975297b77e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38637162-7909-4f8d-a7fa-ed975297b77e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:07:14,253 [237] [DEBUG] [app] Ending request: urn:request:3394b6c0-6901-4688-9264-d92f9ab98c7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3394b6c0-6901-4688-9264-d92f9ab98c7d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:07:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 05:07:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
expiredappspecifictokenworker stdout | 2025-02-07 05:07:15,322 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:07:16,200 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:07:16,243 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:07:17,379 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:07:17,384 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:07:17,387 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:07:17,389 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:07:17,393 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:07:17,461 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:17,848 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:07:18,256 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:07:18,310 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:07:18,411 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:07:18,414 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:07:18,416 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:07:18,543 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:07:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:28.732147+00:00 (in 9.999527 seconds)
notificationworker stdout | 2025-02-07 05:07:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:28 UTC)" (scheduled at 2025-02-07 05:07:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:07:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:07:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 18, 732884), True, datetime.datetime(2025, 2, 7, 5, 7, 18, 732884), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:07:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:07:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:07:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:07:18,920 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:07:19,298 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:07:19,470 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:07:19,729 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:07:19,733 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:07:19,738 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:07:19,741 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:07:19,744 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:07:19,747 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
manifestbackfillworker stdout | 2025-02-07 05:07:19,751 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
gunicorn-registry stdout | 2025-02-07 05:07:19,754 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:07:19,756 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:07:19,759 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
buildlogsarchiver stdout | 2025-02-07 05:07:21,166 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:07:21,332 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:07:21,732 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:07:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:07:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:03.419110+00:00 (in 39.997756 seconds)
repositorygcworker stdout | 2025-02-07 05:07:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:23 UTC)" (scheduled at 2025-02-07 05:07:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:07:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:07:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:07:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:07:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:40.424250+00:00 (in 16.000749 seconds)
gcworker stdout | 2025-02-07 05:07:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:54 UTC)" (scheduled at 2025-02-07 05:07:24.423055+00:00)
gcworker stdout | 2025-02-07 05:07:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:07:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:07:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:07:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:33.814166+00:00 (in 5.081574 seconds)
notificationworker stdout | 2025-02-07 05:07:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:38 UTC)" (scheduled at 2025-02-07 05:07:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:07:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:07:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 28, 732876), True, datetime.datetime(2025, 2, 7, 5, 7, 28, 732876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:07:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:07:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:07:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:07:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:07:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:44.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 05:07:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:44 UTC)" (scheduled at 2025-02-07 05:07:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:07:29,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:07:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:07:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:07:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:07:29,235 [238] [DEBUG] [app] Starting request: urn:request:7c8e3d7e-d67a-4ad4-b9c6-1145ca872928 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:29,235 [235] [DEBUG] [app] Starting request: urn:request:b4a4e074-c915-4f64-9b1e-35677afe7a14 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:07:29,238 [239] [DEBUG] [app] Starting request: urn:request:e7950fbd-69bc-42d4-93e7-afad929d4348 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:29,238 [245] [DEBUG] [app] Starting request: urn:request:9c8800b4-4e3c-4a63-984d-b6541023643a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:29,238 [245] [DEBUG] [app] Ending request: urn:request:9c8800b4-4e3c-4a63-984d-b6541023643a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c8800b4-4e3c-4a63-984d-b6541023643a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:07:29,238 [239] [DEBUG] [app] Ending request: urn:request:e7950fbd-69bc-42d4-93e7-afad929d4348 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e7950fbd-69bc-42d4-93e7-afad929d4348', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:07:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:07:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:07:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:29,243 [237] [DEBUG] [app] Starting request: urn:request:d639b548-fb00-4207-8976-8b27941d98cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:29,243 [236] [DEBUG] [app] Starting request: urn:request:d4fd3e16-cd96-4ffd-a888-289bbed3c8a7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:29,243 [237] [DEBUG] [app] Ending request: urn:request:d639b548-fb00-4207-8976-8b27941d98cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d639b548-fb00-4207-8976-8b27941d98cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:07:29,243 [236] [DEBUG] [app] Ending request: urn:request:d4fd3e16-cd96-4ffd-a888-289bbed3c8a7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4fd3e16-cd96-4ffd-a888-289bbed3c8a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:07:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:29,255 [238] [DEBUG] [app] Ending request: urn:request:7c8e3d7e-d67a-4ad4-b9c6-1145ca872928 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c8e3d7e-d67a-4ad4-b9c6-1145ca872928', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:29,255 [235] [DEBUG] [app] Ending request: urn:request:b4a4e074-c915-4f64-9b1e-35677afe7a14 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b4a4e074-c915-4f64-9b1e-35677afe7a14', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:07:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:07:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:07:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 05:07:30,691 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:07:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:07:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:32.625945+00:00 (in 1.004405 seconds)
namespacegcworker stdout | 2025-02-07 05:07:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:31 UTC)" (scheduled at 2025-02-07 05:07:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:07:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:07:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 31, 621817), True, datetime.datetime(2025, 2, 7, 5, 7, 31, 621817), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:07:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:07:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:07:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:07:32,401 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:07:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:07:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:31.621131+00:00 (in 58.994743 seconds)
namespacegcworker stdout | 2025-02-07 05:07:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:32 UTC)" (scheduled at 2025-02-07 05:07:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:07:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:07:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:07:33,589 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:07:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:38.732147+00:00 (in 4.917553 seconds)
notificationworker stdout | 2025-02-07 05:07:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:12:33 UTC)" (scheduled at 2025-02-07 05:07:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:07:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 7, 33, 814828), 'notification/%'])
notificationworker stdout | 2025-02-07 05:07:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 7, 33, 814828), True, datetime.datetime(2025, 2, 7, 5, 7, 33, 814828), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:07:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 7, 33, 814828), True, datetime.datetime(2025, 2, 7, 5, 7, 33, 814828), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 7, 33, 814828), 'notification/%'])
notificationworker stdout | 2025-02-07 05:07:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:07:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:12:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:07:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:07:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:03.919286+00:00 (in 29.999559 seconds)
buildlogsarchiver stdout | 2025-02-07 05:07:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:03 UTC)" (scheduled at 2025-02-07 05:07:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:07:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 7, 33, 919973), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:07:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:07:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:07:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:56.713547+00:00 (in 18.997187 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:37 UTC)" (scheduled at 2025-02-07 05:07:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:07:37,906 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:07:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:48.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 05:07:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:48 UTC)" (scheduled at 2025-02-07 05:07:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:07:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:07:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 38, 732872), True, datetime.datetime(2025, 2, 7, 5, 7, 38, 732872), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:07:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:07:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:07:38,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:07:38,887 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:07:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:07:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:54.423055+00:00 (in 13.998347 seconds)
gcworker stdout | 2025-02-07 05:07:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:10 UTC)" (scheduled at 2025-02-07 05:07:40.424250+00:00)
gcworker stdout | 2025-02-07 05:07:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:07:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904560431, None, 1, 0])
gcworker stdout | 2025-02-07 05:07:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:07:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:07:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:07:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:11.831521+00:00 (in 29.999508 seconds)
autopruneworker stdout | 2025-02-07 05:07:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:11 UTC)" (scheduled at 2025-02-07 05:07:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:07:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901261836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:07:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:07:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:07:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:07:42,832 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:07:43,390 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:07:43,970 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:07:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:07:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:59.225378+00:00 (in 14.999512 seconds)
quotatotalworker stdout | 2025-02-07 05:07:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:59 UTC)" (scheduled at 2025-02-07 05:07:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:07:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:07:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:07:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:07:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:07:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:07:44,236 [238] [DEBUG] [app] Starting request: urn:request:c647295e-f7d3-4a8a-afc8-6488ee977e09 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:44,236 [235] [DEBUG] [app] Starting request: urn:request:b561d2a8-a167-4663-a185-204e7dfd45bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:07:44,239 [246] [DEBUG] [app] Starting request: urn:request:49a12ff0-0a31-4106-ad3f-33c2681e2478 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:44,239 [245] [DEBUG] [app] Starting request: urn:request:a526a078-9b8b-4d49-802a-a35ba124bb1d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:44,239 [246] [DEBUG] [app] Ending request: urn:request:49a12ff0-0a31-4106-ad3f-33c2681e2478 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49a12ff0-0a31-4106-ad3f-33c2681e2478', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:07:44,239 [245] [DEBUG] [app] Ending request: urn:request:a526a078-9b8b-4d49-802a-a35ba124bb1d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a526a078-9b8b-4d49-802a-a35ba124bb1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:07:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:07:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:07:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:44,243 [238] [DEBUG] [app] Starting request: urn:request:452a25b8-3bf4-43d6-9031-e6b6888b9a23 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:44,243 [237] [DEBUG] [app] Starting request: urn:request:b7ca78e3-d086-4ee2-a3d3-35764d2d6f11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:44,243 [238] [DEBUG] [app] Ending request: urn:request:452a25b8-3bf4-43d6-9031-e6b6888b9a23 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:452a25b8-3bf4-43d6-9031-e6b6888b9a23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:07:44,243 [237] [DEBUG] [app] Ending request: urn:request:b7ca78e3-d086-4ee2-a3d3-35764d2d6f11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7ca78e3-d086-4ee2-a3d3-35764d2d6f11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:07:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:07:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:44,255 [238] [DEBUG] [app] Ending request: urn:request:c647295e-f7d3-4a8a-afc8-6488ee977e09 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c647295e-f7d3-4a8a-afc8-6488ee977e09', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:44,255 [235] [DEBUG] [app] Ending request: urn:request:b561d2a8-a167-4663-a185-204e7dfd45bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b561d2a8-a167-4663-a185-204e7dfd45bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:07:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:07:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:07:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
expiredappspecifictokenworker stdout | 2025-02-07 05:07:45,358 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:07:46,237 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:07:46,266 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:07:47,388 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:07:47,391 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:07:47,394 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:07:47,398 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:07:47,400 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:07:47,497 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:47,884 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:07:48,274 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:07:48,319 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:07:48,418 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:07:48,421 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:07:48,424 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:07:48,580 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:07:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:50.814677+00:00 (in 2.082073 seconds)
notificationworker stdout | 2025-02-07 05:07:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:58 UTC)" (scheduled at 2025-02-07 05:07:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:07:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:07:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 48, 732896), True, datetime.datetime(2025, 2, 7, 5, 7, 48, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:07:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:07:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:07:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:07:58 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:07:48,937 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:07:49,333 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:07:49,479 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:07:49,739 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:07:49,742 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:07:49,748 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:07:49,751 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:07:49,754 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:07:49,756 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:07:49,761 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:07:49,764 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:07:49,768 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:07:49,787 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:07:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:07:58.732147+00:00 (in 7.917028 seconds)
notificationworker stdout | 2025-02-07 05:07:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:50 UTC)" (scheduled at 2025-02-07 05:07:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:07:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:07:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:07:51,203 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:07:51,359 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:07:51,742 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:07:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:07:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:10.424250+00:00 (in 16.000743 seconds)
gcworker stdout | 2025-02-07 05:07:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:24 UTC)" (scheduled at 2025-02-07 05:07:54.423055+00:00)
gcworker stdout | 2025-02-07 05:07:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:07:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:07:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:07:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:56.118855+00:00 (in 59.999531 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:07:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:56 UTC)" (scheduled at 2025-02-07 05:07:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:07:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:07:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:07:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:37.715878+00:00 (in 41.001871 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:56 UTC)" (scheduled at 2025-02-07 05:07:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:07:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:07:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:07:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:08.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 05:07:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:08 UTC)" (scheduled at 2025-02-07 05:07:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:07:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:07:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 7, 58, 732895), True, datetime.datetime(2025, 2, 7, 5, 7, 58, 732895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:07:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:07:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:07:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:07:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:07:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:14.225378+00:00 (in 14.999550 seconds)
quotatotalworker stdout | 2025-02-07 05:07:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:14 UTC)" (scheduled at 2025-02-07 05:07:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:07:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:07:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:07:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:07:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:07:59,235 [237] [DEBUG] [app] Starting request: urn:request:42e10ab9-723e-472c-bcec-44f9765a7066 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:59,235 [236] [DEBUG] [app] Starting request: urn:request:f442cdef-4284-4ca7-832a-4e41f29cfa21 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:07:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:07:59,238 [239] [DEBUG] [app] Starting request: urn:request:28e5f4d1-92c3-4c71-94f4-2fa1d9a23c79 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:59,238 [246] [DEBUG] [app] Starting request: urn:request:932d53c9-dabc-4113-9b5b-357f6e7ada79 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:07:59,238 [239] [DEBUG] [app] Ending request: urn:request:28e5f4d1-92c3-4c71-94f4-2fa1d9a23c79 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28e5f4d1-92c3-4c71-94f4-2fa1d9a23c79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:07:59,238 [246] [DEBUG] [app] Ending request: urn:request:932d53c9-dabc-4113-9b5b-357f6e7ada79 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:932d53c9-dabc-4113-9b5b-357f6e7ada79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:07:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:07:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:07:59,241 [236] [DEBUG] [app] Starting request: urn:request:27b6c3d1-e44f-4a26-ae14-097b7c107bc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:59,242 [235] [DEBUG] [app] Starting request: urn:request:c2e4a25b-98de-478e-9bfb-bdd1d4aac372 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:07:59,242 [236] [DEBUG] [app] Ending request: urn:request:27b6c3d1-e44f-4a26-ae14-097b7c107bc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27b6c3d1-e44f-4a26-ae14-097b7c107bc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:07:59,242 [235] [DEBUG] [app] Ending request: urn:request:c2e4a25b-98de-478e-9bfb-bdd1d4aac372 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2e4a25b-98de-478e-9bfb-bdd1d4aac372', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:07:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:07:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:07:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:07:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:07:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:07:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:07:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:07:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:07:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:07:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:07:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:07:59,254 [236] [DEBUG] [app] Ending request: urn:request:f442cdef-4284-4ca7-832a-4e41f29cfa21 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f442cdef-4284-4ca7-832a-4e41f29cfa21', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:59,254 [237] [DEBUG] [app] Ending request: urn:request:42e10ab9-723e-472c-bcec-44f9765a7066 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42e10ab9-723e-472c-bcec-44f9765a7066', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:07:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:07:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:07:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:07:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:07:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:07:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 05:07:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:07:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:04.936688+00:00 (in 4.998304 seconds)
exportactionlogsworker stdout | 2025-02-07 05:07:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:59 UTC)" (scheduled at 2025-02-07 05:07:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:07:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:07:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:08:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:08:00,715 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:08:02,437 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:08:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:08:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:23.420886+00:00 (in 20.001280 seconds)
repositorygcworker stdout | 2025-02-07 05:08:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:03 UTC)" (scheduled at 2025-02-07 05:08:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:08:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:08:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 3, 419882), True, datetime.datetime(2025, 2, 7, 5, 8, 3, 419882), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:08:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:08:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:08:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:08:03,618 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:08:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:08:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:33.919286+00:00 (in 29.999492 seconds)
buildlogsarchiver stdout | 2025-02-07 05:08:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:33 UTC)" (scheduled at 2025-02-07 05:08:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:08:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 8, 3, 920081), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:08:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:08:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:08:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:08:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:08:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:59.937899+00:00 (in 55.000768 seconds)
exportactionlogsworker stdout | 2025-02-07 05:08:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:04 UTC)" (scheduled at 2025-02-07 05:08:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:08:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:08:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 4, 937439), True, datetime.datetime(2025, 2, 7, 5, 8, 4, 937439), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:08:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:08:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:08:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:08:07,937 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:08:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:08:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:18.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 05:08:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:18 UTC)" (scheduled at 2025-02-07 05:08:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:08:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:08:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 8, 732890), True, datetime.datetime(2025, 2, 7, 5, 8, 8, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:08:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:08:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:08:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:08:08,919 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:08:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:08:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:24.423055+00:00 (in 13.998379 seconds)
gcworker stdout | 2025-02-07 05:08:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:40 UTC)" (scheduled at 2025-02-07 05:08:10.424250+00:00)
gcworker stdout | 2025-02-07 05:08:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:08:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904590431, None, 1, 0])
gcworker stdout | 2025-02-07 05:08:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:08:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:08:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:08:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:41.831521+00:00 (in 29.999574 seconds)
autopruneworker stdout | 2025-02-07 05:08:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:41 UTC)" (scheduled at 2025-02-07 05:08:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:08:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901291836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:08:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:08:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:08:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:08:12,843 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:08:13,426 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:08:14,007 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:08:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:08:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:29.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 05:08:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:29 UTC)" (scheduled at 2025-02-07 05:08:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:08:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:08:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:08:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:08:14,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:08:14,235 [237] [DEBUG] [app] Starting request: urn:request:5306e2f6-695e-4568-bd88-3fdfd27670f0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:14,235 [235] [DEBUG] [app] Starting request: urn:request:61d71b58-a99f-41ec-beab-0c486c3a6f6c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:08:14,238 [239] [DEBUG] [app] Starting request: urn:request:841c0faf-0339-4c83-93f5-0ab24acf5644 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:14,238 [243] [DEBUG] [app] Starting request: urn:request:2bcb0397-91b2-4c64-8fa3-041aac316874 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:14,238 [243] [DEBUG] [app] Ending request: urn:request:2bcb0397-91b2-4c64-8fa3-041aac316874 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2bcb0397-91b2-4c64-8fa3-041aac316874', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:08:14,238 [239] [DEBUG] [app] Ending request: urn:request:841c0faf-0339-4c83-93f5-0ab24acf5644 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:841c0faf-0339-4c83-93f5-0ab24acf5644', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:08:14,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:08:14,238 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:14,241 [235] [DEBUG] [app] Starting request: urn:request:68f496fb-52b1-4a17-8242-455c2e901bc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:08:14,241 [238] [DEBUG] [app] Starting request: urn:request:3196b5f2-0167-4655-8ae6-2e2e74697ca6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:08:14,241 [235] [DEBUG] [app] Ending request: urn:request:68f496fb-52b1-4a17-8242-455c2e901bc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68f496fb-52b1-4a17-8242-455c2e901bc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:08:14,241 [238] [DEBUG] [app] Ending request: urn:request:3196b5f2-0167-4655-8ae6-2e2e74697ca6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3196b5f2-0167-4655-8ae6-2e2e74697ca6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:08:14,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:08:14,241 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:08:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:08:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:08:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:14,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:14,252 [237] [DEBUG] [app] Ending request: urn:request:5306e2f6-695e-4568-bd88-3fdfd27670f0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5306e2f6-695e-4568-bd88-3fdfd27670f0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:14,252 [235] [DEBUG] [app] Ending request: urn:request:61d71b58-a99f-41ec-beab-0c486c3a6f6c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61d71b58-a99f-41ec-beab-0c486c3a6f6c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:08:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:08:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:08:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
expiredappspecifictokenworker stdout | 2025-02-07 05:08:15,395 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:08:16,244 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:08:16,302 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:08:17,395 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:08:17,398 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:08:17,402 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:08:17,405 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:08:17,407 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:08:17,512 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:17,897 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:08:18,310 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:08:18,341 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:08:18,425 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:08:18,428 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:08:18,431 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:08:18,589 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:08:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:08:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:28.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 05:08:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:28 UTC)" (scheduled at 2025-02-07 05:08:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:08:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:08:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 18, 732859), True, datetime.datetime(2025, 2, 7, 5, 8, 18, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:08:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:08:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:08:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:08:18,969 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:08:19,342 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:08:19,490 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:08:19,749 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:08:19,752 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:08:19,755 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:08:19,758 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:08:19,761 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:08:19,768 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:08:19,771 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:08:19,773 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:08:19,776 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:08:19,823 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:08:21,218 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:08:21,390 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:08:21,754 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:08:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:08:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:32.420707+00:00 (in 8.999364 seconds)
repositorygcworker stdout | 2025-02-07 05:08:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:23 UTC)" (scheduled at 2025-02-07 05:08:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:08:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:08:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:08:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:08:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:40.424250+00:00 (in 16.000732 seconds)
gcworker stdout | 2025-02-07 05:08:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:54 UTC)" (scheduled at 2025-02-07 05:08:24.423055+00:00)
gcworker stdout | 2025-02-07 05:08:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:08:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:08:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:08:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:08:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:38.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 05:08:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:38 UTC)" (scheduled at 2025-02-07 05:08:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:08:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:08:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 28, 732904), True, datetime.datetime(2025, 2, 7, 5, 8, 28, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:08:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:08:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:08:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:08:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:08:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:44.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 05:08:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:44 UTC)" (scheduled at 2025-02-07 05:08:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:08:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:08:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:08:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:08:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:08:29,235 [238] [DEBUG] [app] Starting request: urn:request:9aac2ffe-70b7-4b55-adce-d0f888b54502 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:29,235 [235] [DEBUG] [app] Starting request: urn:request:78a927b9-1f5c-4b90-8b30-3d08176b6841 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:08:29,238 [239] [DEBUG] [app] Starting request: urn:request:c1975863-1e57-42d3-92c4-250c304667e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:29,238 [245] [DEBUG] [app] Starting request: urn:request:60f6fbc1-2bee-452c-b501-239cf9d4826c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:29,238 [239] [DEBUG] [app] Ending request: urn:request:c1975863-1e57-42d3-92c4-250c304667e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c1975863-1e57-42d3-92c4-250c304667e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:08:29,238 [245] [DEBUG] [app] Ending request: urn:request:60f6fbc1-2bee-452c-b501-239cf9d4826c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60f6fbc1-2bee-452c-b501-239cf9d4826c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:08:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:08:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:29,241 [237] [DEBUG] [app] Starting request: urn:request:57ec9604-5f08-4e89-bada-53ee6150a3c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:08:29,241 [237] [DEBUG] [app] Ending request: urn:request:57ec9604-5f08-4e89-bada-53ee6150a3c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57ec9604-5f08-4e89-bada-53ee6150a3c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:08:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:29,242 [235] [DEBUG] [app] Starting request: urn:request:e96f792d-714a-46ee-b02a-3ea220b65f87 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:08:29,242 [235] [DEBUG] [app] Ending request: urn:request:e96f792d-714a-46ee-b02a-3ea220b65f87 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e96f792d-714a-46ee-b02a-3ea220b65f87', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:08:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:08:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:08:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:08:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:29,254 [235] [DEBUG] [app] Ending request: urn:request:78a927b9-1f5c-4b90-8b30-3d08176b6841 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78a927b9-1f5c-4b90-8b30-3d08176b6841', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:08:29,254 [238] [DEBUG] [app] Ending request: urn:request:9aac2ffe-70b7-4b55-adce-d0f888b54502 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9aac2ffe-70b7-4b55-adce-d0f888b54502', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:08:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:08:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 05:08:30,724 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:08:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:08:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:32.625945+00:00 (in 1.004369 seconds)
namespacegcworker stdout | 2025-02-07 05:08:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:31 UTC)" (scheduled at 2025-02-07 05:08:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:08:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:08:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 31, 621853), True, datetime.datetime(2025, 2, 7, 5, 8, 31, 621853), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:08:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:08:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:08:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:08:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:08:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:03.419110+00:00 (in 30.997978 seconds)
repositorygcworker stdout | 2025-02-07 05:08:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:13:32 UTC)" (scheduled at 2025-02-07 05:08:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:08:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 8, 32, 421400), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:08:32,427 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 8, 32, 421400), True, datetime.datetime(2025, 2, 7, 5, 8, 32, 421400), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:08:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 8, 32, 421400), True, datetime.datetime(2025, 2, 7, 5, 8, 32, 421400), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 8, 32, 421400), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:08:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:08:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:13:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:08:32,473 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:08:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:08:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:31.621131+00:00 (in 58.994719 seconds)
namespacegcworker stdout | 2025-02-07 05:08:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:32 UTC)" (scheduled at 2025-02-07 05:08:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:08:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:08:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:08:33,641 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:08:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:08:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:03.919286+00:00 (in 29.999515 seconds)
buildlogsarchiver stdout | 2025-02-07 05:08:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:03 UTC)" (scheduled at 2025-02-07 05:08:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:08:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 8, 33, 920023), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:08:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:08:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:08:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:56.713547+00:00 (in 18.997207 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:37 UTC)" (scheduled at 2025-02-07 05:08:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:08:37,969 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:08:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:08:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:48.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 05:08:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:48 UTC)" (scheduled at 2025-02-07 05:08:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:08:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:08:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 38, 732818), True, datetime.datetime(2025, 2, 7, 5, 8, 38, 732818), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:08:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:08:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:08:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:08:38,957 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:08:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:08:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:54.423055+00:00 (in 13.998379 seconds)
gcworker stdout | 2025-02-07 05:08:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:10 UTC)" (scheduled at 2025-02-07 05:08:40.424250+00:00)
gcworker stdout | 2025-02-07 05:08:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:08:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904620431, None, 1, 0])
gcworker stdout | 2025-02-07 05:08:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:08:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:08:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:08:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:11.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 05:08:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:11 UTC)" (scheduled at 2025-02-07 05:08:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:08:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901321836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:08:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:08:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:08:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:08:42,874 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:08:43,461 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:08:44,042 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:08:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:08:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:59.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 05:08:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:59 UTC)" (scheduled at 2025-02-07 05:08:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:08:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:08:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:08:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:08:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:08:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:08:44,235 [237] [DEBUG] [app] Starting request: urn:request:f465a646-a819-4566-9621-4e81526b587a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:44,235 [235] [DEBUG] [app] Starting request: urn:request:8e92fcf9-164d-4b6d-a3b7-551d05868286 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:08:44,239 [239] [DEBUG] [app] Starting request: urn:request:1ec8e60e-3c2e-48ca-b5fe-c0cb8685b405 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:44,239 [246] [DEBUG] [app] Starting request: urn:request:3e74cba5-ec53-4aa3-b4d3-982463a770d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:44,240 [239] [DEBUG] [app] Ending request: urn:request:1ec8e60e-3c2e-48ca-b5fe-c0cb8685b405 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1ec8e60e-3c2e-48ca-b5fe-c0cb8685b405', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:08:44,240 [246] [DEBUG] [app] Ending request: urn:request:3e74cba5-ec53-4aa3-b4d3-982463a770d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3e74cba5-ec53-4aa3-b4d3-982463a770d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:08:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:08:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:44,243 [236] [DEBUG] [app] Starting request: urn:request:7ac4610d-757e-4940-91bd-a51eaef6a54b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:08:44,244 [238] [DEBUG] [app] Starting request: urn:request:0fdfcf2d-1c71-4d9f-b02a-f615176d692f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:08:44,244 [236] [DEBUG] [app] Ending request: urn:request:7ac4610d-757e-4940-91bd-a51eaef6a54b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ac4610d-757e-4940-91bd-a51eaef6a54b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:08:44,244 [238] [DEBUG] [app] Ending request: urn:request:0fdfcf2d-1c71-4d9f-b02a-f615176d692f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0fdfcf2d-1c71-4d9f-b02a-f615176d692f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:08:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:44,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:08:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:08:44,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:44,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:44,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:44,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:44,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:44,256 [237] [DEBUG] [app] Ending request: urn:request:f465a646-a819-4566-9621-4e81526b587a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f465a646-a819-4566-9621-4e81526b587a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:44,257 [235] [DEBUG] [app] Ending request: urn:request:8e92fcf9-164d-4b6d-a3b7-551d05868286 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8e92fcf9-164d-4b6d-a3b7-551d05868286', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:44,257 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:08:44,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:08:44,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:08:44,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
expiredappspecifictokenworker stdout | 2025-02-07 05:08:45,434 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:08:46,266 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:08:46,338 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:08:47,404 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:08:47,408 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:08:47,411 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:08:47,414 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:08:47,417 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:08:47,546 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:47,930 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:08:48,346 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:08:48,378 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:08:48,434 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:08:48,437 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:08:48,439 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:08:48,618 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:08:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:08:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:50.814677+00:00 (in 2.082050 seconds)
notificationworker stdout | 2025-02-07 05:08:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:58 UTC)" (scheduled at 2025-02-07 05:08:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:08:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:08:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 48, 732950), True, datetime.datetime(2025, 2, 7, 5, 8, 48, 732950), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:08:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:08:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:08:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:08:58 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:08:48,994 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:08:49,378 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:08:49,527 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:08:49,758 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:08:49,761 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:08:49,764 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:08:49,770 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:08:49,773 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:08:49,777 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:08:49,780 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:08:49,783 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:08:49,786 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:08:49,831 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:08:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:08:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:08:58.732147+00:00 (in 7.916988 seconds)
notificationworker stdout | 2025-02-07 05:08:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:50 UTC)" (scheduled at 2025-02-07 05:08:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:08:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:08:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:08:51,254 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:08:51,415 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:08:51,790 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:08:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:08:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:10.424250+00:00 (in 16.000708 seconds)
gcworker stdout | 2025-02-07 05:08:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:24 UTC)" (scheduled at 2025-02-07 05:08:54.423055+00:00)
gcworker stdout | 2025-02-07 05:08:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:08:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:08:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:08:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:56.118855+00:00 (in 59.999481 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:08:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:56 UTC)" (scheduled at 2025-02-07 05:08:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:08:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:08:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:08:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:37.715878+00:00 (in 41.001881 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:56 UTC)" (scheduled at 2025-02-07 05:08:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:08:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:08:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:08:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:08.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 05:08:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:08 UTC)" (scheduled at 2025-02-07 05:08:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:08:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:08:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 8, 58, 732926), True, datetime.datetime(2025, 2, 7, 5, 8, 58, 732926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:08:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:08:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:08:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:08:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:08:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:14.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 05:08:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:14 UTC)" (scheduled at 2025-02-07 05:08:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:08:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:08:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:08:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:08:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:08:59,235 [235] [DEBUG] [app] Starting request: urn:request:87641332-178b-40d4-9290-b81cb5e88852 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:59,235 [238] [DEBUG] [app] Starting request: urn:request:0b746468-e10d-4377-8b88-4e5914f4cc9b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:08:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:08:59,238 [239] [DEBUG] [app] Starting request: urn:request:fd72134c-489a-47c4-aae4-f788cbaf04e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:59,238 [246] [DEBUG] [app] Starting request: urn:request:6758b2c8-4f2b-4f85-8ff3-e19e56323c69 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:08:59,238 [239] [DEBUG] [app] Ending request: urn:request:fd72134c-489a-47c4-aae4-f788cbaf04e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd72134c-489a-47c4-aae4-f788cbaf04e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:08:59,239 [246] [DEBUG] [app] Ending request: urn:request:6758b2c8-4f2b-4f85-8ff3-e19e56323c69 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6758b2c8-4f2b-4f85-8ff3-e19e56323c69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:08:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:08:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:08:59,241 [235] [DEBUG] [app] Starting request: urn:request:dabfb324-d4a2-4f11-bddb-97a9d2c097bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:08:59,241 [235] [DEBUG] [app] Ending request: urn:request:dabfb324-d4a2-4f11-bddb-97a9d2c097bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dabfb324-d4a2-4f11-bddb-97a9d2c097bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:08:59,241 [237] [DEBUG] [app] Starting request: urn:request:302f56fd-72f8-4b34-837e-81b7c364e4e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.000 159 0.001)
gunicorn-web stdout | 2025-02-07 05:08:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:59,242 [237] [DEBUG] [app] Ending request: urn:request:302f56fd-72f8-4b34-837e-81b7c364e4e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:302f56fd-72f8-4b34-837e-81b7c364e4e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:08:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:08:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:08:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:08:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:59,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:08:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:08:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:08:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:08:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:08:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:08:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:08:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:59,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:08:59,253 [235] [DEBUG] [app] Ending request: urn:request:87641332-178b-40d4-9290-b81cb5e88852 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:87641332-178b-40d4-9290-b81cb5e88852', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:59,253 [238] [DEBUG] [app] Ending request: urn:request:0b746468-e10d-4377-8b88-4e5914f4cc9b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b746468-e10d-4377-8b88-4e5914f4cc9b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:08:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:08:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:08:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:08:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:08:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:08:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
exportactionlogsworker stdout | 2025-02-07 05:08:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:08:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:04.936688+00:00 (in 4.998290 seconds)
exportactionlogsworker stdout | 2025-02-07 05:08:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:59 UTC)" (scheduled at 2025-02-07 05:08:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:08:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:08:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:09:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:09:00,758 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:09:02,509 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:09:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:09:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:23.420886+00:00 (in 20.001331 seconds)
repositorygcworker stdout | 2025-02-07 05:09:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:03 UTC)" (scheduled at 2025-02-07 05:09:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:09:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:09:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 3, 419841), True, datetime.datetime(2025, 2, 7, 5, 9, 3, 419841), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:09:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:09:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:09:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:09:03,677 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:09:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:09:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:33.919286+00:00 (in 29.999566 seconds)
buildlogsarchiver stdout | 2025-02-07 05:09:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:33 UTC)" (scheduled at 2025-02-07 05:09:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:09:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 9, 3, 919973), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:09:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:09:03,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:09:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:09:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:09:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:52.937648+00:00 (in 48.000537 seconds)
exportactionlogsworker stdout | 2025-02-07 05:09:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:04 UTC)" (scheduled at 2025-02-07 05:09:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:09:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:09:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 4, 937411), True, datetime.datetime(2025, 2, 7, 5, 9, 4, 937411), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:09:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:09:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:09:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:09:07,988 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:09:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:09:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:18.732147+00:00 (in 9.999440 seconds)
notificationworker stdout | 2025-02-07 05:09:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:18 UTC)" (scheduled at 2025-02-07 05:09:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:09:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:09:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 8, 733010), True, datetime.datetime(2025, 2, 7, 5, 9, 8, 733010), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:09:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:09:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:09:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:09:08,975 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:09:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:09:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:24.423055+00:00 (in 13.998387 seconds)
gcworker stdout | 2025-02-07 05:09:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:40 UTC)" (scheduled at 2025-02-07 05:09:10.424250+00:00)
gcworker stdout | 2025-02-07 05:09:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:09:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904650431, None, 1, 0])
gcworker stdout | 2025-02-07 05:09:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:09:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:09:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:09:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:41.831521+00:00 (in 29.999561 seconds)
autopruneworker stdout | 2025-02-07 05:09:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:41 UTC)" (scheduled at 2025-02-07 05:09:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:09:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901351836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:09:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:09:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:09:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:09:12,907 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:09:13,497 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:09:14,078 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:09:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:09:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:29.225378+00:00 (in 14.999505 seconds)
quotatotalworker stdout | 2025-02-07 05:09:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:29 UTC)" (scheduled at 2025-02-07 05:09:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:09:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:09:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:09:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:09:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:09:14,236 [235] [DEBUG] [app] Starting request: urn:request:52dc07c2-edc1-4a9c-a334-cbb504528edc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:14,236 [237] [DEBUG] [app] Starting request: urn:request:756ae7f8-a68e-4db5-a1b1-f5f4ee4e528a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:09:14,239 [239] [DEBUG] [app] Starting request: urn:request:897a1e62-3585-416a-9478-e2a0ea9a382a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:14,239 [246] [DEBUG] [app] Starting request: urn:request:81630b07-ae84-495a-9950-5649d81d3b89 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:14,239 [239] [DEBUG] [app] Ending request: urn:request:897a1e62-3585-416a-9478-e2a0ea9a382a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:897a1e62-3585-416a-9478-e2a0ea9a382a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:14,239 [246] [DEBUG] [app] Ending request: urn:request:81630b07-ae84-495a-9950-5649d81d3b89 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81630b07-ae84-495a-9950-5649d81d3b89', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:09:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:09:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:14,242 [235] [DEBUG] [app] Starting request: urn:request:9ffaf8d2-c775-4e35-ac3b-d2ceb64408b7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:14,242 [235] [DEBUG] [app] Ending request: urn:request:9ffaf8d2-c775-4e35-ac3b-d2ceb64408b7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9ffaf8d2-c775-4e35-ac3b-d2ceb64408b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:09:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:09:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:14,243 [238] [DEBUG] [app] Starting request: urn:request:6f8c37ba-b06f-4a6b-885e-5bcd84c1f277 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:14,243 [238] [DEBUG] [app] Ending request: urn:request:6f8c37ba-b06f-4a6b-885e-5bcd84c1f277 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f8c37ba-b06f-4a6b-885e-5bcd84c1f277', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:09:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:09:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:14,253 [235] [DEBUG] [app] Ending request: urn:request:52dc07c2-edc1-4a9c-a334-cbb504528edc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:52dc07c2-edc1-4a9c-a334-cbb504528edc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:09:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:09:14,254 [237] [DEBUG] [app] Ending request: urn:request:756ae7f8-a68e-4db5-a1b1-f5f4ee4e528a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:756ae7f8-a68e-4db5-a1b1-f5f4ee4e528a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:09:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
expiredappspecifictokenworker stdout | 2025-02-07 05:09:15,447 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:09:16,298 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:09:16,348 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:09:17,412 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:09:17,418 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:09:17,421 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:09:17,423 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:09:17,426 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:09:17,583 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:17,941 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:09:18,373 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:09:18,415 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:09:18,442 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:09:18,444 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:09:18,447 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:09:18,655 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:09:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:09:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:28.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 05:09:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:28 UTC)" (scheduled at 2025-02-07 05:09:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:09:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:09:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 18, 732851), True, datetime.datetime(2025, 2, 7, 5, 9, 18, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:09:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:09:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:09:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:09:19,012 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:09:19,393 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:09:19,563 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:09:19,767 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:09:19,773 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:09:19,777 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:09:19,780 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:09:19,782 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:09:19,786 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:09:19,789 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:09:19,792 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:09:19,795 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:09:19,854 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:09:21,283 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:09:21,451 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:09:21,821 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:09:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:09:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:03.419110+00:00 (in 39.997756 seconds)
repositorygcworker stdout | 2025-02-07 05:09:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:23 UTC)" (scheduled at 2025-02-07 05:09:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:09:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:09:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:09:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:09:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:40.424250+00:00 (in 16.000761 seconds)
gcworker stdout | 2025-02-07 05:09:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:54 UTC)" (scheduled at 2025-02-07 05:09:24.423055+00:00)
gcworker stdout | 2025-02-07 05:09:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:09:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:09:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:09:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:09:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:38.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 05:09:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:38 UTC)" (scheduled at 2025-02-07 05:09:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:09:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:09:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 28, 732854), True, datetime.datetime(2025, 2, 7, 5, 9, 28, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:09:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:09:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:09:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:09:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:09:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:44.225378+00:00 (in 14.999501 seconds)
quotatotalworker stdout | 2025-02-07 05:09:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:44 UTC)" (scheduled at 2025-02-07 05:09:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:09:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:09:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:09:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:09:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:09:29,235 [238] [DEBUG] [app] Starting request: urn:request:ceb721b6-194e-4252-b288-9c241a73d384 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:29,235 [237] [DEBUG] [app] Starting request: urn:request:35c78d6f-42b5-46f1-90ca-5a592b03eaf9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:09:29,238 [246] [DEBUG] [app] Starting request: urn:request:2a19e39a-f198-41af-8da6-7d382b0a3aae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:29,238 [245] [DEBUG] [app] Starting request: urn:request:9d64a202-2261-4069-846f-1fda986360b7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:29,239 [246] [DEBUG] [app] Ending request: urn:request:2a19e39a-f198-41af-8da6-7d382b0a3aae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a19e39a-f198-41af-8da6-7d382b0a3aae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:29,239 [245] [DEBUG] [app] Ending request: urn:request:9d64a202-2261-4069-846f-1fda986360b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d64a202-2261-4069-846f-1fda986360b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:09:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:09:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:29,241 [235] [DEBUG] [app] Starting request: urn:request:ae323c49-dec1-438b-bd6e-3f657999634b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:29,241 [236] [DEBUG] [app] Starting request: urn:request:7171f419-ab61-4fe4-908f-6f0481eb66c3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:29,242 [235] [DEBUG] [app] Ending request: urn:request:ae323c49-dec1-438b-bd6e-3f657999634b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae323c49-dec1-438b-bd6e-3f657999634b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:09:29,242 [236] [DEBUG] [app] Ending request: urn:request:7171f419-ab61-4fe4-908f-6f0481eb66c3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7171f419-ab61-4fe4-908f-6f0481eb66c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:09:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:09:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:09:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:29,253 [238] [DEBUG] [app] Ending request: urn:request:ceb721b6-194e-4252-b288-9c241a73d384 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ceb721b6-194e-4252-b288-9c241a73d384', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:29,253 [237] [DEBUG] [app] Ending request: urn:request:35c78d6f-42b5-46f1-90ca-5a592b03eaf9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:35c78d6f-42b5-46f1-90ca-5a592b03eaf9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:09:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:09:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:09:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
storagereplication stdout | 2025-02-07 05:09:30,774 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:09:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:09:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:32.625945+00:00 (in 1.004367 seconds)
namespacegcworker stdout | 2025-02-07 05:09:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:31 UTC)" (scheduled at 2025-02-07 05:09:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:09:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:09:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 31, 621891), True, datetime.datetime(2025, 2, 7, 5, 9, 31, 621891), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:09:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:09:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:09:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:09:32,518 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:09:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:09:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:31.621131+00:00 (in 58.994725 seconds)
namespacegcworker stdout | 2025-02-07 05:09:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:32 UTC)" (scheduled at 2025-02-07 05:09:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:09:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:09:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:09:33,707 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:09:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:09:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:03.919286+00:00 (in 29.999514 seconds)
buildlogsarchiver stdout | 2025-02-07 05:09:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:03 UTC)" (scheduled at 2025-02-07 05:09:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:09:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 9, 33, 920063), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:09:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:09:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:09:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:56.713547+00:00 (in 18.997203 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:37 UTC)" (scheduled at 2025-02-07 05:09:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:09:38,000 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:09:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:09:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:48.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 05:09:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:48 UTC)" (scheduled at 2025-02-07 05:09:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:09:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:09:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 38, 732853), True, datetime.datetime(2025, 2, 7, 5, 9, 38, 732853), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:09:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:09:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:09:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:09:39,011 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:09:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:09:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:54.423055+00:00 (in 13.998314 seconds)
gcworker stdout | 2025-02-07 05:09:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:10 UTC)" (scheduled at 2025-02-07 05:09:40.424250+00:00)
gcworker stdout | 2025-02-07 05:09:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:09:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904680431, None, 1, 0])
gcworker stdout | 2025-02-07 05:09:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:09:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:09:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:09:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:11.831521+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2025-02-07 05:09:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:11 UTC)" (scheduled at 2025-02-07 05:09:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:09:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901381836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:09:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:09:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:09:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:09:42,925 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:09:43,520 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:09:44,093 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:09:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:09:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:59.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 05:09:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:59 UTC)" (scheduled at 2025-02-07 05:09:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:09:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:09:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:09:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:09:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:09:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:09:44,236 [235] [DEBUG] [app] Starting request: urn:request:ee10848f-a265-4e57-b9c8-cdabc917acef (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:44,236 [236] [DEBUG] [app] Starting request: urn:request:304ee4cd-0ae6-42a7-89f4-f808b896b4bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:09:44,238 [239] [DEBUG] [app] Starting request: urn:request:0e2b08a8-2c0c-4cea-b0da-20deda062068 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:44,238 [243] [DEBUG] [app] Starting request: urn:request:136d8066-8272-4b19-bc42-0dc567a660f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:44,239 [239] [DEBUG] [app] Ending request: urn:request:0e2b08a8-2c0c-4cea-b0da-20deda062068 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e2b08a8-2c0c-4cea-b0da-20deda062068', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:44,239 [243] [DEBUG] [app] Ending request: urn:request:136d8066-8272-4b19-bc42-0dc567a660f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:136d8066-8272-4b19-bc42-0dc567a660f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:09:44,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:09:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:44,241 [235] [DEBUG] [app] Starting request: urn:request:be456845-0b2f-4de2-9d01-ae17ce6b4972 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:44,242 [235] [DEBUG] [app] Ending request: urn:request:be456845-0b2f-4de2-9d01-ae17ce6b4972 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be456845-0b2f-4de2-9d01-ae17ce6b4972', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:09:44,242 [238] [DEBUG] [app] Starting request: urn:request:fdf796cd-84fb-468d-8192-f1435bc3ae4e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:44,242 [238] [DEBUG] [app] Ending request: urn:request:fdf796cd-84fb-468d-8192-f1435bc3ae4e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdf796cd-84fb-468d-8192-f1435bc3ae4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:09:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:09:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:09:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:44,253 [235] [DEBUG] [app] Ending request: urn:request:ee10848f-a265-4e57-b9c8-cdabc917acef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee10848f-a265-4e57-b9c8-cdabc917acef', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:44,253 [236] [DEBUG] [app] Ending request: urn:request:304ee4cd-0ae6-42a7-89f4-f808b896b4bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:304ee4cd-0ae6-42a7-89f4-f808b896b4bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:09:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:09:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:09:44,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
expiredappspecifictokenworker stdout | 2025-02-07 05:09:45,483 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:09:46,334 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:09:46,362 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:09:47,421 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:09:47,425 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:09:47,429 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:09:47,432 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:09:47,434 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:09:47,607 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:47,968 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:09:48,407 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:09:48,446 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-secscan stdout | 2025-02-07 05:09:48,449 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:09:48,453 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:09:48,455 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
exportactionlogsworker stdout | 2025-02-07 05:09:48,666 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:09:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:09:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:50.814677+00:00 (in 2.082080 seconds)
notificationworker stdout | 2025-02-07 05:09:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:58 UTC)" (scheduled at 2025-02-07 05:09:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:09:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:09:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 48, 732805), True, datetime.datetime(2025, 2, 7, 5, 9, 48, 732805), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:09:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:09:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:09:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:09:58 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:09:49,048 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:09:49,429 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:09:49,597 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:09:49,776 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:09:49,786 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:09:49,794 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:09:49,797 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:09:49,800 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:09:49,803 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:09:49,807 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:09:49,810 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:09:49,813 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:09:49,890 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:09:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:09:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:58.732147+00:00 (in 7.916994 seconds)
notificationworker stdout | 2025-02-07 05:09:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:50 UTC)" (scheduled at 2025-02-07 05:09:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:09:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:09:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:09:51,309 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:09:51,484 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:09:51,843 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
exportactionlogsworker stdout | 2025-02-07 05:09:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:09:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:09:59.937899+00:00 (in 6.999801 seconds)
exportactionlogsworker stdout | 2025-02-07 05:09:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:14:52 UTC)" (scheduled at 2025-02-07 05:09:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:09:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 9, 52, 938390), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:09:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 9, 52, 938390), True, datetime.datetime(2025, 2, 7, 5, 9, 52, 938390), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:09:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 9, 52, 938390), True, datetime.datetime(2025, 2, 7, 5, 9, 52, 938390), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 9, 52, 938390), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:09:52,949 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:09:52,949 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:14:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:09:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:09:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:10.424250+00:00 (in 16.000709 seconds)
gcworker stdout | 2025-02-07 05:09:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:24 UTC)" (scheduled at 2025-02-07 05:09:54.423055+00:00)
gcworker stdout | 2025-02-07 05:09:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:09:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:09:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:09:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:56.118855+00:00 (in 59.999508 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:09:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:56 UTC)" (scheduled at 2025-02-07 05:09:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:09:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:09:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:09:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:37.715878+00:00 (in 41.001887 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:56 UTC)" (scheduled at 2025-02-07 05:09:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:09:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:09:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:09:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:08.732147+00:00 (in 9.999434 seconds)
notificationworker stdout | 2025-02-07 05:09:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:08 UTC)" (scheduled at 2025-02-07 05:09:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:09:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:09:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 9, 58, 733013), True, datetime.datetime(2025, 2, 7, 5, 9, 58, 733013), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:09:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:09:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:09:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:09:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:09:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:14.225378+00:00 (in 14.999568 seconds)
quotatotalworker stdout | 2025-02-07 05:09:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:14 UTC)" (scheduled at 2025-02-07 05:09:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:09:59,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:09:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:09:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:09:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:09:59,236 [235] [DEBUG] [app] Starting request: urn:request:e46a4c1b-2cac-4e83-a522-7f516e2b0123 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:59,236 [238] [DEBUG] [app] Starting request: urn:request:0102807d-803a-4132-8309-47382a577d05 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:09:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:09:59,239 [245] [DEBUG] [app] Starting request: urn:request:f174187f-15a8-45f6-b2f9-c9ebd4b1c1f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:59,239 [243] [DEBUG] [app] Starting request: urn:request:2647e4a3-4736-412c-9014-5c63a8d6307b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:09:59,239 [245] [DEBUG] [app] Ending request: urn:request:f174187f-15a8-45f6-b2f9-c9ebd4b1c1f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f174187f-15a8-45f6-b2f9-c9ebd4b1c1f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:59,239 [243] [DEBUG] [app] Ending request: urn:request:2647e4a3-4736-412c-9014-5c63a8d6307b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2647e4a3-4736-412c-9014-5c63a8d6307b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:09:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:09:59,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:09:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:09:59,242 [236] [DEBUG] [app] Starting request: urn:request:043189cd-eb18-4eb1-a08f-41cef2a88830 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:59,242 [237] [DEBUG] [app] Starting request: urn:request:c24ae322-91d1-4456-87b5-59a993906239 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:09:59,242 [236] [DEBUG] [app] Ending request: urn:request:043189cd-eb18-4eb1-a08f-41cef2a88830 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:043189cd-eb18-4eb1-a08f-41cef2a88830', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:09:59,242 [237] [DEBUG] [app] Ending request: urn:request:c24ae322-91d1-4456-87b5-59a993906239 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c24ae322-91d1-4456-87b5-59a993906239', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:09:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:09:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:09:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:09:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:09:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:09:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:09:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:09:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:09:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:09:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:09:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:09:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:09:59,254 [235] [DEBUG] [app] Ending request: urn:request:e46a4c1b-2cac-4e83-a522-7f516e2b0123 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e46a4c1b-2cac-4e83-a522-7f516e2b0123', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:59,254 [238] [DEBUG] [app] Ending request: urn:request:0102807d-803a-4132-8309-47382a577d05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0102807d-803a-4132-8309-47382a577d05', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:09:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:09:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:09:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:09:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:09:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:09:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:09:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:09:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:04.936688+00:00 (in 4.998332 seconds)
exportactionlogsworker stdout | 2025-02-07 05:09:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:59 UTC)" (scheduled at 2025-02-07 05:09:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:09:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:09:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:10:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:10:00,781 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:10:02,552 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:10:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:10:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:23.420886+00:00 (in 20.001307 seconds)
repositorygcworker stdout | 2025-02-07 05:10:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:03 UTC)" (scheduled at 2025-02-07 05:10:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:10:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:10:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 3, 419883), True, datetime.datetime(2025, 2, 7, 5, 10, 3, 419883), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:10:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:10:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:10:03,428 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:10:03,719 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:10:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:10:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:33.919286+00:00 (in 29.999483 seconds)
buildlogsarchiver stdout | 2025-02-07 05:10:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:33 UTC)" (scheduled at 2025-02-07 05:10:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:10:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 10, 3, 920047), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:10:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:10:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:10:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:10:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:10:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:59.937899+00:00 (in 55.000791 seconds)
exportactionlogsworker stdout | 2025-02-07 05:10:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:04 UTC)" (scheduled at 2025-02-07 05:10:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:10:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:10:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 4, 937400), True, datetime.datetime(2025, 2, 7, 5, 10, 4, 937400), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:10:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:10:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:10:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:10:08,017 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:10:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:10:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:18.732147+00:00 (in 9.999521 seconds)
notificationworker stdout | 2025-02-07 05:10:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:18 UTC)" (scheduled at 2025-02-07 05:10:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:10:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:10:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 8, 732827), True, datetime.datetime(2025, 2, 7, 5, 10, 8, 732827), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:10:08,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:10:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:10:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:10:09,047 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:10:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:10:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:24.423055+00:00 (in 13.998377 seconds)
gcworker stdout | 2025-02-07 05:10:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:40 UTC)" (scheduled at 2025-02-07 05:10:10.424250+00:00)
gcworker stdout | 2025-02-07 05:10:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:10:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904710431, None, 1, 0])
gcworker stdout | 2025-02-07 05:10:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:10:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:10:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:10:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:41.831521+00:00 (in 29.999516 seconds)
autopruneworker stdout | 2025-02-07 05:10:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:41 UTC)" (scheduled at 2025-02-07 05:10:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:10:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901411836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:10:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:10:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:10:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:10:12,961 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:10:13,531 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:10:14,129 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:10:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:10:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:29.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 05:10:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:29 UTC)" (scheduled at 2025-02-07 05:10:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:10:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:10:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:10:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:10:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:10:14,236 [238] [DEBUG] [app] Starting request: urn:request:1d9674fd-1414-4105-9d93-2b8f86dd021c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:14,236 [235] [DEBUG] [app] Starting request: urn:request:2a96bfbe-889b-47d1-8962-831c5b4eee47 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:10:14,240 [239] [DEBUG] [app] Starting request: urn:request:86d036d3-9a1e-4a3a-aeaa-86e2f4c0d68d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:14,240 [246] [DEBUG] [app] Starting request: urn:request:feaa1c7e-e8d3-40e7-ba12-3a149328e801 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:14,240 [239] [DEBUG] [app] Ending request: urn:request:86d036d3-9a1e-4a3a-aeaa-86e2f4c0d68d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86d036d3-9a1e-4a3a-aeaa-86e2f4c0d68d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:10:14,240 [246] [DEBUG] [app] Ending request: urn:request:feaa1c7e-e8d3-40e7-ba12-3a149328e801 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:feaa1c7e-e8d3-40e7-ba12-3a149328e801', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:10:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:10:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:10:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:14,243 [238] [DEBUG] [app] Starting request: urn:request:d4329ffb-4846-47eb-807c-8af34f967c20 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:14,243 [238] [DEBUG] [app] Ending request: urn:request:d4329ffb-4846-47eb-807c-8af34f967c20 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4329ffb-4846-47eb-807c-8af34f967c20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:10:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:10:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:14,244 [236] [DEBUG] [app] Starting request: urn:request:484c479a-5f3d-4af6-b06b-3d0dae55f7ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:14,244 [236] [DEBUG] [app] Ending request: urn:request:484c479a-5f3d-4af6-b06b-3d0dae55f7ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:484c479a-5f3d-4af6-b06b-3d0dae55f7ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:10:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:10:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:14,254 [238] [DEBUG] [app] Ending request: urn:request:1d9674fd-1414-4105-9d93-2b8f86dd021c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1d9674fd-1414-4105-9d93-2b8f86dd021c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:10:14,256 [235] [DEBUG] [app] Ending request: urn:request:2a96bfbe-889b-47d1-8962-831c5b4eee47 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a96bfbe-889b-47d1-8962-831c5b4eee47', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
expiredappspecifictokenworker stdout | 2025-02-07 05:10:15,518 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:10:16,371 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:10:16,374 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:10:17,430 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:10:17,434 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:10:17,436 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:10:17,439 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:10:17,442 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:10:17,643 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:18,005 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:10:18,443 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-secscan stdout | 2025-02-07 05:10:18,457 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:10:18,459 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:10:18,462 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
servicekey stdout | 2025-02-07 05:10:18,481 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:10:18,703 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:10:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:10:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:28.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 05:10:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:28 UTC)" (scheduled at 2025-02-07 05:10:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:10:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:10:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 18, 732796), True, datetime.datetime(2025, 2, 7, 5, 10, 18, 732796), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:10:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:10:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:10:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:28 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:10:19,079 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:10:19,460 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:10:19,629 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:10:19,797 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:10:19,800 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:10:19,805 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:10:19,809 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:10:19,811 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:10:19,814 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:10:19,817 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:10:19,819 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:10:19,823 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:10:19,926 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:10:21,322 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:10:21,520 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:10:21,871 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:10:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:10:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:03.419110+00:00 (in 39.997747 seconds)
repositorygcworker stdout | 2025-02-07 05:10:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:23 UTC)" (scheduled at 2025-02-07 05:10:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:10:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:10:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:10:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:10:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:40.424250+00:00 (in 16.000759 seconds)
gcworker stdout | 2025-02-07 05:10:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:54 UTC)" (scheduled at 2025-02-07 05:10:24.423055+00:00)
gcworker stdout | 2025-02-07 05:10:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:10:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:10:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:10:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:10:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:38.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 05:10:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:38 UTC)" (scheduled at 2025-02-07 05:10:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:10:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:10:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 28, 732801), True, datetime.datetime(2025, 2, 7, 5, 10, 28, 732801), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:10:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:10:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:10:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:10:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:10:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:44.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 05:10:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:44 UTC)" (scheduled at 2025-02-07 05:10:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:10:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:10:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:10:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:10:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:10:29,235 [237] [DEBUG] [app] Starting request: urn:request:2d914c80-6084-4284-8057-abe5b2b6fdf9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:29,235 [235] [DEBUG] [app] Starting request: urn:request:cef666be-daa5-4487-9ba6-d69b3c3a3045 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:10:29,238 [239] [DEBUG] [app] Starting request: urn:request:eeb17d0e-8515-40c0-bbad-8704791c068c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:29,238 [246] [DEBUG] [app] Starting request: urn:request:8865f25f-4313-4b91-b6a5-9bc7e9a4e114 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:29,238 [239] [DEBUG] [app] Ending request: urn:request:eeb17d0e-8515-40c0-bbad-8704791c068c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eeb17d0e-8515-40c0-bbad-8704791c068c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:10:29,239 [246] [DEBUG] [app] Ending request: urn:request:8865f25f-4313-4b91-b6a5-9bc7e9a4e114 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8865f25f-4313-4b91-b6a5-9bc7e9a4e114', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:10:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:10:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:29,241 [236] [DEBUG] [app] Starting request: urn:request:d008067d-61b7-48de-bb66-f6e37f071611 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:29,242 [236] [DEBUG] [app] Ending request: urn:request:d008067d-61b7-48de-bb66-f6e37f071611 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d008067d-61b7-48de-bb66-f6e37f071611', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:10:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:10:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:29,242 [238] [DEBUG] [app] Starting request: urn:request:99ed2424-a7e5-412a-b688-28a5db4d9ec4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:29,243 [238] [DEBUG] [app] Ending request: urn:request:99ed2424-a7e5-412a-b688-28a5db4d9ec4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:99ed2424-a7e5-412a-b688-28a5db4d9ec4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:10:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:10:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:29,253 [235] [DEBUG] [app] Ending request: urn:request:cef666be-daa5-4487-9ba6-d69b3c3a3045 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cef666be-daa5-4487-9ba6-d69b3c3a3045', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 05:10:29,254 [237] [DEBUG] [app] Ending request: urn:request:2d914c80-6084-4284-8057-abe5b2b6fdf9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d914c80-6084-4284-8057-abe5b2b6fdf9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 05:10:30,794 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:10:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:10:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:32.625945+00:00 (in 1.004373 seconds)
namespacegcworker stdout | 2025-02-07 05:10:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:31 UTC)" (scheduled at 2025-02-07 05:10:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:10:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:10:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 31, 621844), True, datetime.datetime(2025, 2, 7, 5, 10, 31, 621844), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:10:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:10:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:10:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:31 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:10:32,584 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
namespacegcworker stdout | 2025-02-07 05:10:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:10:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:31.621131+00:00 (in 58.994691 seconds)
namespacegcworker stdout | 2025-02-07 05:10:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:32 UTC)" (scheduled at 2025-02-07 05:10:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:10:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:10:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:32 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:10:33,755 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:10:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:10:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:03.919286+00:00 (in 29.999567 seconds)
buildlogsarchiver stdout | 2025-02-07 05:10:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:03 UTC)" (scheduled at 2025-02-07 05:10:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:10:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 10, 33, 919957), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:10:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:10:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:10:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:56.713547+00:00 (in 18.997054 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:37 UTC)" (scheduled at 2025-02-07 05:10:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:37,724 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:37,724 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:37,725 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:10:38,056 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:10:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:10:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:48.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:10:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:48 UTC)" (scheduled at 2025-02-07 05:10:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:10:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:10:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 38, 732861), True, datetime.datetime(2025, 2, 7, 5, 10, 38, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:10:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:10:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:10:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:10:39,083 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:10:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:10:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:54.423055+00:00 (in 13.998395 seconds)
gcworker stdout | 2025-02-07 05:10:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:10 UTC)" (scheduled at 2025-02-07 05:10:40.424250+00:00)
gcworker stdout | 2025-02-07 05:10:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:10:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904740430, None, 1, 0])
gcworker stdout | 2025-02-07 05:10:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:10:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:10:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:10:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:11.831521+00:00 (in 29.999528 seconds)
autopruneworker stdout | 2025-02-07 05:10:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:11 UTC)" (scheduled at 2025-02-07 05:10:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:10:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901441836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:10:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:10:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:10:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:10:42,998 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:10:43,543 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:10:44,165 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:10:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:10:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:59.225378+00:00 (in 14.999571 seconds)
quotatotalworker stdout | 2025-02-07 05:10:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:59 UTC)" (scheduled at 2025-02-07 05:10:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:10:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:10:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:10:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:10:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:10:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:10:44,235 [235] [DEBUG] [app] Starting request: urn:request:2da267b3-46db-48d9-a1d3-d33458b44ac5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:44,235 [238] [DEBUG] [app] Starting request: urn:request:3f6c0219-15e2-4e47-b892-163140fa1515 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:10:44,239 [239] [DEBUG] [app] Starting request: urn:request:a9239550-d87a-4216-9be6-03e9475470a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:44,239 [246] [DEBUG] [app] Starting request: urn:request:4283879d-8ce1-4c3c-9246-18c93b06388c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:44,239 [239] [DEBUG] [app] Ending request: urn:request:a9239550-d87a-4216-9be6-03e9475470a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a9239550-d87a-4216-9be6-03e9475470a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:10:44,240 [246] [DEBUG] [app] Ending request: urn:request:4283879d-8ce1-4c3c-9246-18c93b06388c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4283879d-8ce1-4c3c-9246-18c93b06388c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:10:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:10:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:44,243 [236] [DEBUG] [app] Starting request: urn:request:7ae9b15a-efe9-4432-a3c7-ff4625c1e241 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:44,243 [237] [DEBUG] [app] Starting request: urn:request:209f35ef-5b40-4aea-b2cd-0a933d0ee2da (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:44,244 [236] [DEBUG] [app] Ending request: urn:request:7ae9b15a-efe9-4432-a3c7-ff4625c1e241 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ae9b15a-efe9-4432-a3c7-ff4625c1e241', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:10:44,244 [237] [DEBUG] [app] Ending request: urn:request:209f35ef-5b40-4aea-b2cd-0a933d0ee2da (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:209f35ef-5b40-4aea-b2cd-0a933d0ee2da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:10:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:10:44,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:44,255 [238] [DEBUG] [app] Ending request: urn:request:3f6c0219-15e2-4e47-b892-163140fa1515 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f6c0219-15e2-4e47-b892-163140fa1515', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:44,255 [235] [DEBUG] [app] Ending request: urn:request:2da267b3-46db-48d9-a1d3-d33458b44ac5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2da267b3-46db-48d9-a1d3-d33458b44ac5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:10:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
expiredappspecifictokenworker stdout | 2025-02-07 05:10:45,530 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:10:46,386 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:10:46,410 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:10:47,440 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:10:47,444 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:10:47,448 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:10:47,450 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:10:47,453 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:10:47,674 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:48,042 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:10:48,466 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:10:48,470 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:10:48,472 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:10:48,478 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:10:48,507 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:10:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:10:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:50.814677+00:00 (in 2.082055 seconds)
notificationworker stdout | 2025-02-07 05:10:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:58 UTC)" (scheduled at 2025-02-07 05:10:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:10:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:10:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 48, 732898), True, datetime.datetime(2025, 2, 7, 5, 10, 48, 732898), 0, 'notification/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:10:48,739 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:10:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:10:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:10:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:10:58 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:10:49,108 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:10:49,495 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:10:49,666 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:10:49,809 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:10:49,812 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:10:49,816 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:10:49,819 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:10:49,822 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:10:49,826 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:10:49,829 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:10:49,831 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:10:49,834 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
manifestbackfillworker stdout | 2025-02-07 05:10:49,934 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:10:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:10:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:10:58.732147+00:00 (in 7.917009 seconds)
notificationworker stdout | 2025-02-07 05:10:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:50 UTC)" (scheduled at 2025-02-07 05:10:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:10:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:10:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:10:51,358 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:10:51,532 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:10:51,908 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:10:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:10:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:10.424250+00:00 (in 16.000686 seconds)
gcworker stdout | 2025-02-07 05:10:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:24 UTC)" (scheduled at 2025-02-07 05:10:54.423055+00:00)
gcworker stdout | 2025-02-07 05:10:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:10:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:10:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:10:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:56.118855+00:00 (in 59.999481 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:10:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:56 UTC)" (scheduled at 2025-02-07 05:10:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:10:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:10:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:10:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:37.715878+00:00 (in 41.001843 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:56 UTC)" (scheduled at 2025-02-07 05:10:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:10:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:10:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:10:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:08.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 05:10:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:08 UTC)" (scheduled at 2025-02-07 05:10:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:10:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:10:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 10, 58, 732867), True, datetime.datetime(2025, 2, 7, 5, 10, 58, 732867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:10:58,747 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:10:58,747 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:10:58,747 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:10:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:10:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:14.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 05:10:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:14 UTC)" (scheduled at 2025-02-07 05:10:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:10:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:10:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:10:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:10:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:10:59,235 [235] [DEBUG] [app] Starting request: urn:request:dcc84306-6935-4190-8dfa-533a1ee22eb8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:59,235 [236] [DEBUG] [app] Starting request: urn:request:39ea3b78-dd87-4521-be7c-8f9399ca7e65 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:10:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:10:59,238 [245] [DEBUG] [app] Starting request: urn:request:a71be761-612a-461d-b74b-83d4297a43fd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:59,238 [239] [DEBUG] [app] Starting request: urn:request:c44f7706-856d-4710-86b5-2355018e3b23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:10:59,239 [239] [DEBUG] [app] Ending request: urn:request:c44f7706-856d-4710-86b5-2355018e3b23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c44f7706-856d-4710-86b5-2355018e3b23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:10:59,239 [245] [DEBUG] [app] Ending request: urn:request:a71be761-612a-461d-b74b-83d4297a43fd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a71be761-612a-461d-b74b-83d4297a43fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:10:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:10:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:10:59,242 [235] [DEBUG] [app] Starting request: urn:request:02323efe-ce60-4a61-a070-e3ce94e6b28c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:59,242 [237] [DEBUG] [app] Starting request: urn:request:e94d2adc-521f-4b61-a3af-d9c2cb329802 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:10:59,242 [235] [DEBUG] [app] Ending request: urn:request:02323efe-ce60-4a61-a070-e3ce94e6b28c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02323efe-ce60-4a61-a070-e3ce94e6b28c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:10:59,242 [237] [DEBUG] [app] Ending request: urn:request:e94d2adc-521f-4b61-a3af-d9c2cb329802 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e94d2adc-521f-4b61-a3af-d9c2cb329802', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:10:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:10:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:10:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:10:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:10:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:10:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:10:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:10:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:10:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:10:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:10:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:10:59,255 [235] [DEBUG] [app] Ending request: urn:request:dcc84306-6935-4190-8dfa-533a1ee22eb8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcc84306-6935-4190-8dfa-533a1ee22eb8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:59,255 [236] [DEBUG] [app] Ending request: urn:request:39ea3b78-dd87-4521-be7c-8f9399ca7e65 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39ea3b78-dd87-4521-be7c-8f9399ca7e65', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:10:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:10:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:10:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:10:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:10:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 05:10:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:10:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:04.936688+00:00 (in 4.998322 seconds)
exportactionlogsworker stdout | 2025-02-07 05:10:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:59 UTC)" (scheduled at 2025-02-07 05:10:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:10:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:10:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:11:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:11:00,830 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:11:02,620 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:11:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:11:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:23.420886+00:00 (in 20.001330 seconds)
repositorygcworker stdout | 2025-02-07 05:11:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:03 UTC)" (scheduled at 2025-02-07 05:11:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:11:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:11:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 3, 419829), True, datetime.datetime(2025, 2, 7, 5, 11, 3, 419829), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:11:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:11:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:11:03,426 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:11:03,784 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:11:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:11:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:33.919286+00:00 (in 29.999565 seconds)
buildlogsarchiver stdout | 2025-02-07 05:11:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:33 UTC)" (scheduled at 2025-02-07 05:11:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:11:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 11, 3, 919972), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:11:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:11:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:11:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:11:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:11:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:59.937899+00:00 (in 55.000768 seconds)
exportactionlogsworker stdout | 2025-02-07 05:11:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:04 UTC)" (scheduled at 2025-02-07 05:11:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:11:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:11:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 4, 937439), True, datetime.datetime(2025, 2, 7, 5, 11, 4, 937439), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:11:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:11:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:11:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:11:08,079 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:11:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:11:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:18.732147+00:00 (in 9.999574 seconds)
notificationworker stdout | 2025-02-07 05:11:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:18 UTC)" (scheduled at 2025-02-07 05:11:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:11:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:11:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 8, 732856), True, datetime.datetime(2025, 2, 7, 5, 11, 8, 732856), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:11:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:11:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:11:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:11:09,098 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:11:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:11:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:24.423055+00:00 (in 13.998291 seconds)
gcworker stdout | 2025-02-07 05:11:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:40 UTC)" (scheduled at 2025-02-07 05:11:10.424250+00:00)
gcworker stdout | 2025-02-07 05:11:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:11:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904770430, None, 1, 0])
gcworker stdout | 2025-02-07 05:11:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:11:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:11:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:11:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:41.831521+00:00 (in 29.999550 seconds)
autopruneworker stdout | 2025-02-07 05:11:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:41 UTC)" (scheduled at 2025-02-07 05:11:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:11:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901471836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:11:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:11:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:11:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:11:13,034 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:11:13,579 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:11:14,202 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:11:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:11:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:29.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 05:11:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:29 UTC)" (scheduled at 2025-02-07 05:11:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:11:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:11:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:11:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:11:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:11:14,236 [235] [DEBUG] [app] Starting request: urn:request:f0c857ca-8c8b-477b-8d29-b7500004735d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:14,236 [238] [DEBUG] [app] Starting request: urn:request:a4f94ce0-fc14-47f7-b778-58b8985023bf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:14,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:11:14,239 [246] [DEBUG] [app] Starting request: urn:request:34b249c9-3eaf-4bca-9e14-08e178cbf505 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:14,239 [244] [DEBUG] [app] Starting request: urn:request:d51a8e89-82d4-4cdd-961c-84ca40aa03ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:14,240 [246] [DEBUG] [app] Ending request: urn:request:34b249c9-3eaf-4bca-9e14-08e178cbf505 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34b249c9-3eaf-4bca-9e14-08e178cbf505', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:11:14,240 [244] [DEBUG] [app] Ending request: urn:request:d51a8e89-82d4-4cdd-961c-84ca40aa03ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d51a8e89-82d4-4cdd-961c-84ca40aa03ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:11:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:11:14,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:11:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:14,242 [238] [DEBUG] [app] Starting request: urn:request:e72f5aff-7e26-4539-94ec-1b382dc70ea8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:11:14,243 [237] [DEBUG] [app] Starting request: urn:request:5f25b107-b59f-42be-9f08-e3192a07e7ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:11:14,243 [238] [DEBUG] [app] Ending request: urn:request:e72f5aff-7e26-4539-94ec-1b382dc70ea8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e72f5aff-7e26-4539-94ec-1b382dc70ea8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:11:14,243 [237] [DEBUG] [app] Ending request: urn:request:5f25b107-b59f-42be-9f08-e3192a07e7ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f25b107-b59f-42be-9f08-e3192a07e7ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:11:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:11:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:14,255 [235] [DEBUG] [app] Ending request: urn:request:f0c857ca-8c8b-477b-8d29-b7500004735d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0c857ca-8c8b-477b-8d29-b7500004735d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:11:14,255 [238] [DEBUG] [app] Ending request: urn:request:a4f94ce0-fc14-47f7-b778-58b8985023bf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a4f94ce0-fc14-47f7-b778-58b8985023bf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:11:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:11:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:11:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:11:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 05:11:15,566 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:11:16,410 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:11:16,446 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:11:17,449 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:11:17,452 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:11:17,455 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:11:17,458 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:11:17,461 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:11:17,710 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:18,049 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:11:18,475 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:11:18,479 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:11:18,482 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:11:18,514 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:11:18,543 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:11:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:11:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:28.732147+00:00 (in 9.999531 seconds)
notificationworker stdout | 2025-02-07 05:11:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:28 UTC)" (scheduled at 2025-02-07 05:11:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:11:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:11:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 18, 732911), True, datetime.datetime(2025, 2, 7, 5, 11, 18, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:11:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:11:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:11:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:28 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:11:18,769 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:11:19,129 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:11:19,517 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:11:19,674 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:11:19,821 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:11:19,824 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:11:19,828 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:11:19,833 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:11:19,836 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:11:19,839 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:11:19,842 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:11:19,845 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:11:19,848 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:11:19,971 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:11:21,394 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:11:21,569 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:11:21,937 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:11:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:11:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:03.419110+00:00 (in 39.997753 seconds)
repositorygcworker stdout | 2025-02-07 05:11:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:23 UTC)" (scheduled at 2025-02-07 05:11:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:11:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:11:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:11:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:11:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:40.424250+00:00 (in 16.000719 seconds)
gcworker stdout | 2025-02-07 05:11:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:54 UTC)" (scheduled at 2025-02-07 05:11:24.423055+00:00)
gcworker stdout | 2025-02-07 05:11:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:11:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:11:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:11:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:11:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:38.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:11:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:38 UTC)" (scheduled at 2025-02-07 05:11:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:11:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:11:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 28, 732849), True, datetime.datetime(2025, 2, 7, 5, 11, 28, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:11:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:11:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:11:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:11:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:11:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:44.225378+00:00 (in 14.999550 seconds)
quotatotalworker stdout | 2025-02-07 05:11:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:44 UTC)" (scheduled at 2025-02-07 05:11:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:11:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:11:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:11:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:11:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:11:29,235 [237] [DEBUG] [app] Starting request: urn:request:069014e7-a662-416a-bb3c-602d1eb6a284 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:29,235 [235] [DEBUG] [app] Starting request: urn:request:32fe2ce2-257a-4f63-838a-148cf13d22f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:11:29,238 [239] [DEBUG] [app] Starting request: urn:request:0f108587-e24d-4c50-82ce-f5df554ecbd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:29,238 [245] [DEBUG] [app] Starting request: urn:request:49c7217b-5256-49e9-947f-47b1ad34532f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:29,239 [239] [DEBUG] [app] Ending request: urn:request:0f108587-e24d-4c50-82ce-f5df554ecbd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f108587-e24d-4c50-82ce-f5df554ecbd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:11:29,239 [245] [DEBUG] [app] Ending request: urn:request:49c7217b-5256-49e9-947f-47b1ad34532f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49c7217b-5256-49e9-947f-47b1ad34532f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:11:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:11:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:11:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:29,242 [236] [DEBUG] [app] Starting request: urn:request:1d0e3f4e-4adc-49de-82c9-f64581e2eaea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:11:29,242 [236] [DEBUG] [app] Ending request: urn:request:1d0e3f4e-4adc-49de-82c9-f64581e2eaea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d0e3f4e-4adc-49de-82c9-f64581e2eaea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:11:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:29,242 [235] [DEBUG] [app] Starting request: urn:request:3367c8ca-9a2e-445e-9725-4de9114b332f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:11:29,243 [235] [DEBUG] [app] Ending request: urn:request:3367c8ca-9a2e-445e-9725-4de9114b332f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3367c8ca-9a2e-445e-9725-4de9114b332f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:11:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:11:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:29,254 [237] [DEBUG] [app] Ending request: urn:request:069014e7-a662-416a-bb3c-602d1eb6a284 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:069014e7-a662-416a-bb3c-602d1eb6a284', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:11:29,254 [235] [DEBUG] [app] Ending request: urn:request:32fe2ce2-257a-4f63-838a-148cf13d22f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:32fe2ce2-257a-4f63-838a-148cf13d22f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:11:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:11:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:11:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:11:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 05:11:30,859 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:11:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:11:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:32.625945+00:00 (in 1.004365 seconds)
namespacegcworker stdout | 2025-02-07 05:11:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:31 UTC)" (scheduled at 2025-02-07 05:11:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:11:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:11:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 31, 621847), True, datetime.datetime(2025, 2, 7, 5, 11, 31, 621847), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:11:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:11:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:11:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:11:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:11:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:13.624301+00:00 (in 40.997911 seconds)
namespacegcworker stdout | 2025-02-07 05:11:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:32 UTC)" (scheduled at 2025-02-07 05:11:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:11:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:11:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:11:32,656 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:11:33,820 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:11:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:11:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:03.919286+00:00 (in 29.999552 seconds)
buildlogsarchiver stdout | 2025-02-07 05:11:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:03 UTC)" (scheduled at 2025-02-07 05:11:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:11:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 11, 33, 919969), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:11:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:11:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:11:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:56.713547+00:00 (in 18.997191 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:37 UTC)" (scheduled at 2025-02-07 05:11:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:11:38,116 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:11:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:11:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:48.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:11:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:48 UTC)" (scheduled at 2025-02-07 05:11:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:11:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:11:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 38, 732843), True, datetime.datetime(2025, 2, 7, 5, 11, 38, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:11:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:11:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:11:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:11:39,135 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:11:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:11:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:54.423055+00:00 (in 13.998358 seconds)
gcworker stdout | 2025-02-07 05:11:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:10 UTC)" (scheduled at 2025-02-07 05:11:40.424250+00:00)
gcworker stdout | 2025-02-07 05:11:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:11:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904800431, None, 1, 0])
gcworker stdout | 2025-02-07 05:11:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:11:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:11:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:11:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:11.831521+00:00 (in 29.999515 seconds)
autopruneworker stdout | 2025-02-07 05:11:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:11 UTC)" (scheduled at 2025-02-07 05:11:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:11:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901501836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:11:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:11:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:11:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:11:43,070 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:11:43,606 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:11:44,211 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
quotatotalworker stdout | 2025-02-07 05:11:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:11:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:59.225378+00:00 (in 14.999568 seconds)
quotatotalworker stdout | 2025-02-07 05:11:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:59 UTC)" (scheduled at 2025-02-07 05:11:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:11:44,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:11:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:11:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:11:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:11:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:11:44,235 [238] [DEBUG] [app] Starting request: urn:request:185867d5-b744-4234-abc8-b9ed1057a1e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:44,235 [235] [DEBUG] [app] Starting request: urn:request:58240d88-4e5a-4a72-8554-a785079b71ba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:11:44,239 [239] [DEBUG] [app] Starting request: urn:request:94428b0d-33b3-42a0-8504-a5829d8af77b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:44,239 [246] [DEBUG] [app] Starting request: urn:request:00d9e865-6fba-4355-a110-5f9e543ebd9c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:44,239 [239] [DEBUG] [app] Ending request: urn:request:94428b0d-33b3-42a0-8504-a5829d8af77b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:94428b0d-33b3-42a0-8504-a5829d8af77b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:11:44,239 [246] [DEBUG] [app] Ending request: urn:request:00d9e865-6fba-4355-a110-5f9e543ebd9c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:00d9e865-6fba-4355-a110-5f9e543ebd9c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:11:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:11:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:44,243 [236] [DEBUG] [app] Starting request: urn:request:b450359a-44fc-4279-ae92-6e150bad8311 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:11:44,243 [237] [DEBUG] [app] Starting request: urn:request:00e0ef4e-b442-4418-b08b-da7eec412b89 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:11:44,243 [236] [DEBUG] [app] Ending request: urn:request:b450359a-44fc-4279-ae92-6e150bad8311 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b450359a-44fc-4279-ae92-6e150bad8311', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:11:44,243 [237] [DEBUG] [app] Ending request: urn:request:00e0ef4e-b442-4418-b08b-da7eec412b89 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:00e0ef4e-b442-4418-b08b-da7eec412b89', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:11:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:11:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:44,255 [235] [DEBUG] [app] Ending request: urn:request:58240d88-4e5a-4a72-8554-a785079b71ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:58240d88-4e5a-4a72-8554-a785079b71ba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:11:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:11:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:11:44,255 [238] [DEBUG] [app] Ending request: urn:request:185867d5-b744-4234-abc8-b9ed1057a1e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:185867d5-b744-4234-abc8-b9ed1057a1e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:11:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:11:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
expiredappspecifictokenworker stdout | 2025-02-07 05:11:45,603 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:11:46,438 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:11:46,483 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:11:47,459 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:11:47,462 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:11:47,466 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:11:47,469 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:11:47,472 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:11:47,735 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:48,056 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:11:48,484 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:11:48,488 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:11:48,492 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:11:48,549 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:11:48,579 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:11:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:11:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:50.814677+00:00 (in 2.082072 seconds)
notificationworker stdout | 2025-02-07 05:11:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:58 UTC)" (scheduled at 2025-02-07 05:11:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:11:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:11:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 48, 732880), True, datetime.datetime(2025, 2, 7, 5, 11, 48, 732880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:11:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:11:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:11:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:11:58 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:11:48,790 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:11:49,166 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:11:49,550 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:11:49,711 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:11:49,831 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:11:49,834 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:11:49,837 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:11:49,840 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:11:49,844 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:11:49,847 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:11:49,850 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:11:49,852 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:11:49,857 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:11:49,994 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:11:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:11:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:11:58.732147+00:00 (in 7.917011 seconds)
notificationworker stdout | 2025-02-07 05:11:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:50 UTC)" (scheduled at 2025-02-07 05:11:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:11:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:11:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:11:51,407 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:11:51,606 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:11:51,974 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:11:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:11:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:10.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 05:11:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:24 UTC)" (scheduled at 2025-02-07 05:11:54.423055+00:00)
gcworker stdout | 2025-02-07 05:11:54,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 05:11:54,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:11:54,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:11:54,429 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:11:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:11:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:56.118855+00:00 (in 59.999492 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:11:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:56 UTC)" (scheduled at 2025-02-07 05:11:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:11:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:11:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:11:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:37.715878+00:00 (in 41.001900 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:56 UTC)" (scheduled at 2025-02-07 05:11:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:11:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:11:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:11:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:08.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 05:11:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:08 UTC)" (scheduled at 2025-02-07 05:11:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:11:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:11:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 11, 58, 732913), True, datetime.datetime(2025, 2, 7, 5, 11, 58, 732913), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:11:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:11:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:11:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:11:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:11:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:14.225378+00:00 (in 14.999530 seconds)
quotatotalworker stdout | 2025-02-07 05:11:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:14 UTC)" (scheduled at 2025-02-07 05:11:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:11:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:11:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:11:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:11:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:11:59,235 [238] [DEBUG] [app] Starting request: urn:request:df37ea70-f71a-46a5-8127-05edbc5258c2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:59,235 [235] [DEBUG] [app] Starting request: urn:request:bcf9bf56-5aa0-4dfd-a069-d15debaf79fd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:11:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:11:59,239 [239] [DEBUG] [app] Starting request: urn:request:fd31cca4-d86c-4546-960d-2bdef1abc3f2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:59,239 [245] [DEBUG] [app] Starting request: urn:request:46bb3bd6-813e-447a-93bd-5542e03b5f04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:11:59,239 [239] [DEBUG] [app] Ending request: urn:request:fd31cca4-d86c-4546-960d-2bdef1abc3f2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd31cca4-d86c-4546-960d-2bdef1abc3f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:11:59,239 [245] [DEBUG] [app] Ending request: urn:request:46bb3bd6-813e-447a-93bd-5542e03b5f04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:46bb3bd6-813e-447a-93bd-5542e03b5f04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:11:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:11:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:11:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:11:59,242 [235] [DEBUG] [app] Starting request: urn:request:db99443e-d14c-48ee-a861-759389561661 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:11:59,242 [238] [DEBUG] [app] Starting request: urn:request:d241ed16-2e76-454b-8978-6489ff565728 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:11:59,242 [235] [DEBUG] [app] Ending request: urn:request:db99443e-d14c-48ee-a861-759389561661 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db99443e-d14c-48ee-a861-759389561661', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:11:59,242 [238] [DEBUG] [app] Ending request: urn:request:d241ed16-2e76-454b-8978-6489ff565728 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d241ed16-2e76-454b-8978-6489ff565728', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:11:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:11:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:11:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:11:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:11:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:11:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:11:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:11:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:11:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:11:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:11:59,254 [238] [DEBUG] [app] Ending request: urn:request:df37ea70-f71a-46a5-8127-05edbc5258c2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df37ea70-f71a-46a5-8127-05edbc5258c2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:11:59,254 [235] [DEBUG] [app] Ending request: urn:request:bcf9bf56-5aa0-4dfd-a069-d15debaf79fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bcf9bf56-5aa0-4dfd-a069-d15debaf79fd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:11:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:11:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:11:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:11:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:11:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:11:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:11:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:11:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:04.936688+00:00 (in 4.998295 seconds)
exportactionlogsworker stdout | 2025-02-07 05:11:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:59 UTC)" (scheduled at 2025-02-07 05:11:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:11:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:11:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:12:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:12:00,896 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:12:02,693 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:12:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:12:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:23.420886+00:00 (in 20.001366 seconds)
repositorygcworker stdout | 2025-02-07 05:12:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:03 UTC)" (scheduled at 2025-02-07 05:12:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:12:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:12:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 3, 419788), True, datetime.datetime(2025, 2, 7, 5, 12, 3, 419788), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:12:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:12:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:12:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:12:03,853 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:12:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:12:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:33.919286+00:00 (in 29.999558 seconds)
buildlogsarchiver stdout | 2025-02-07 05:12:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:33 UTC)" (scheduled at 2025-02-07 05:12:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:12:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 12, 3, 919973), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:12:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:12:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:12:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:12:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:12:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:59.937899+00:00 (in 55.000782 seconds)
exportactionlogsworker stdout | 2025-02-07 05:12:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:04 UTC)" (scheduled at 2025-02-07 05:12:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:12:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:12:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 4, 937452), True, datetime.datetime(2025, 2, 7, 5, 12, 4, 937452), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:12:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:12:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:12:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:12:08,137 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:12:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:18.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 05:12:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:18 UTC)" (scheduled at 2025-02-07 05:12:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:12:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:12:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 8, 732905), True, datetime.datetime(2025, 2, 7, 5, 12, 8, 732905), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:12:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:12:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:12:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:12:09,152 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:12:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:12:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:24.423055+00:00 (in 13.998344 seconds)
gcworker stdout | 2025-02-07 05:12:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:40 UTC)" (scheduled at 2025-02-07 05:12:10.424250+00:00)
gcworker stdout | 2025-02-07 05:12:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:12:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904830431, None, 1, 0])
gcworker stdout | 2025-02-07 05:12:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:12:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:12:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:12:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:41.831521+00:00 (in 29.999493 seconds)
autopruneworker stdout | 2025-02-07 05:12:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:41 UTC)" (scheduled at 2025-02-07 05:12:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:12:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901531836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:12:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:12:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:12:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:12:13,106 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
namespacegcworker stdout | 2025-02-07 05:12:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:12:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:31.621131+00:00 (in 17.996389 seconds)
namespacegcworker stdout | 2025-02-07 05:12:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:17:13 UTC)" (scheduled at 2025-02-07 05:12:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:12:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 12, 13, 624988), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:12:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 12, 13, 624988), True, datetime.datetime(2025, 2, 7, 5, 12, 13, 624988), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:12:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 12, 13, 624988), True, datetime.datetime(2025, 2, 7, 5, 12, 13, 624988), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 12, 13, 624988), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:12:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:12:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:17:13 UTC)" executed successfully
builder stdout | 2025-02-07 05:12:13,642 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:12:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:12:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:29.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 05:12:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:29 UTC)" (scheduled at 2025-02-07 05:12:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:12:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:12:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:12:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:12:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:12:14,235 [237] [DEBUG] [app] Starting request: urn:request:8a9d65cf-3bef-4215-a3c4-b34970b3889b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:14,236 [235] [DEBUG] [app] Starting request: urn:request:feb7a9ed-8e2a-4632-b4b4-1fcc02f970b5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
logrotateworker stdout | 2025-02-07 05:12:14,239 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
gunicorn-registry stdout | 2025-02-07 05:12:14,239 [239] [DEBUG] [app] Starting request: urn:request:18ab76e4-5e6d-44cb-8254-e7c524ce8976 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:14,239 [246] [DEBUG] [app] Starting request: urn:request:5d6d9f4a-6a54-4b37-98e7-c8939f9a1a5c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:14,239 [239] [DEBUG] [app] Ending request: urn:request:18ab76e4-5e6d-44cb-8254-e7c524ce8976 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18ab76e4-5e6d-44cb-8254-e7c524ce8976', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:12:14,239 [246] [DEBUG] [app] Ending request: urn:request:5d6d9f4a-6a54-4b37-98e7-c8939f9a1a5c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d6d9f4a-6a54-4b37-98e7-c8939f9a1a5c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:12:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:12:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:12:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:14,243 [238] [DEBUG] [app] Starting request: urn:request:7ed85b33-ccfd-45d9-a326-7abf0285fc99 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:14,243 [236] [DEBUG] [app] Starting request: urn:request:7783ce60-49c8-4568-a0d8-e1318d36d89a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:14,243 [238] [DEBUG] [app] Ending request: urn:request:7ed85b33-ccfd-45d9-a326-7abf0285fc99 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ed85b33-ccfd-45d9-a326-7abf0285fc99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:12:14,243 [236] [DEBUG] [app] Ending request: urn:request:7783ce60-49c8-4568-a0d8-e1318d36d89a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7783ce60-49c8-4568-a0d8-e1318d36d89a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:12:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:12:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:12:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:14,255 [237] [DEBUG] [app] Ending request: urn:request:8a9d65cf-3bef-4215-a3c4-b34970b3889b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a9d65cf-3bef-4215-a3c4-b34970b3889b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:14,255 [235] [DEBUG] [app] Ending request: urn:request:feb7a9ed-8e2a-4632-b4b4-1fcc02f970b5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:feb7a9ed-8e2a-4632-b4b4-1fcc02f970b5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:12:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:12:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:12:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
expiredappspecifictokenworker stdout | 2025-02-07 05:12:15,639 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:12:16,474 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:12:16,490 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:12:17,468 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:12:17,471 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:12:17,474 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:12:17,477 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:12:17,480 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:12:17,754 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:18,088 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:12:18,493 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:12:18,497 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:12:18,499 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:12:18,578 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:12:18,603 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:12:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:28.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 05:12:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:28 UTC)" (scheduled at 2025-02-07 05:12:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:12:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:12:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 18, 732851), True, datetime.datetime(2025, 2, 7, 5, 12, 18, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:12:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:12:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:12:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:28 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:12:18,827 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:12:19,202 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:12:19,583 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:12:19,747 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:12:19,841 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:12:19,844 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:12:19,846 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:12:19,850 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:12:19,853 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:12:19,856 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:12:19,859 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:12:19,862 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:12:19,866 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:12:20,026 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:12:21,442 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:12:21,640 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:12:22,010 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:12:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:12:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:03.419110+00:00 (in 39.997751 seconds)
repositorygcworker stdout | 2025-02-07 05:12:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:23 UTC)" (scheduled at 2025-02-07 05:12:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:12:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:12:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:12:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:12:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:40.424250+00:00 (in 16.000739 seconds)
gcworker stdout | 2025-02-07 05:12:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:54 UTC)" (scheduled at 2025-02-07 05:12:24.423055+00:00)
gcworker stdout | 2025-02-07 05:12:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:12:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:12:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:12:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:33.814166+00:00 (in 5.081578 seconds)
notificationworker stdout | 2025-02-07 05:12:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:38 UTC)" (scheduled at 2025-02-07 05:12:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:12:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:12:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 28, 732898), True, datetime.datetime(2025, 2, 7, 5, 12, 28, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:12:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:12:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:12:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:12:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:12:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:44.225378+00:00 (in 14.999572 seconds)
quotatotalworker stdout | 2025-02-07 05:12:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:44 UTC)" (scheduled at 2025-02-07 05:12:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:12:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:12:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:12:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:12:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:12:29,236 [235] [DEBUG] [app] Starting request: urn:request:d9d35f39-b321-4418-a922-99fb9dfd9f17 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:29,236 [238] [DEBUG] [app] Starting request: urn:request:ba3cdb4c-0cb1-4a00-b01b-be64de2aab32 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:12:29,239 [239] [DEBUG] [app] Starting request: urn:request:18b498a8-3529-412f-91c4-87df3d09160a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:29,239 [245] [DEBUG] [app] Starting request: urn:request:f816d964-573a-45c0-93ca-8b19c9a25b60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:29,239 [239] [DEBUG] [app] Ending request: urn:request:18b498a8-3529-412f-91c4-87df3d09160a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18b498a8-3529-412f-91c4-87df3d09160a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:12:29,239 [245] [DEBUG] [app] Ending request: urn:request:f816d964-573a-45c0-93ca-8b19c9a25b60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f816d964-573a-45c0-93ca-8b19c9a25b60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:12:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:12:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:29,242 [236] [DEBUG] [app] Starting request: urn:request:d1579a29-ee49-4376-92a3-8a89504ca1ac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:29,242 [235] [DEBUG] [app] Starting request: urn:request:17507174-6bac-4f01-b156-b1f83c5d32f1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:29,242 [236] [DEBUG] [app] Ending request: urn:request:d1579a29-ee49-4376-92a3-8a89504ca1ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d1579a29-ee49-4376-92a3-8a89504ca1ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:12:29,242 [235] [DEBUG] [app] Ending request: urn:request:17507174-6bac-4f01-b156-b1f83c5d32f1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17507174-6bac-4f01-b156-b1f83c5d32f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:12:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:12:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:29,254 [238] [DEBUG] [app] Ending request: urn:request:ba3cdb4c-0cb1-4a00-b01b-be64de2aab32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba3cdb4c-0cb1-4a00-b01b-be64de2aab32', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:29,254 [235] [DEBUG] [app] Ending request: urn:request:d9d35f39-b321-4418-a922-99fb9dfd9f17 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9d35f39-b321-4418-a922-99fb9dfd9f17', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:12:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:12:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:12:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
storagereplication stdout | 2025-02-07 05:12:30,914 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:12:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:12:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:32.625945+00:00 (in 1.004370 seconds)
namespacegcworker stdout | 2025-02-07 05:12:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:31 UTC)" (scheduled at 2025-02-07 05:12:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:12:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:12:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 31, 621846), True, datetime.datetime(2025, 2, 7, 5, 12, 31, 621846), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:12:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:12:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:12:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:12:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:12:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:31.621131+00:00 (in 58.994735 seconds)
namespacegcworker stdout | 2025-02-07 05:12:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:32 UTC)" (scheduled at 2025-02-07 05:12:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:12:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:12:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:12:32,719 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 05:12:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:38.732147+00:00 (in 4.917550 seconds)
notificationworker stdout | 2025-02-07 05:12:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:17:33 UTC)" (scheduled at 2025-02-07 05:12:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:12:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 12, 33, 814871), 'notification/%'])
notificationworker stdout | 2025-02-07 05:12:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 12, 33, 814871), True, datetime.datetime(2025, 2, 7, 5, 12, 33, 814871), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:12:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 12, 33, 814871), True, datetime.datetime(2025, 2, 7, 5, 12, 33, 814871), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 12, 33, 814871), 'notification/%'])
notificationworker stdout | 2025-02-07 05:12:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:12:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:17:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:12:33,865 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:12:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:12:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:03.919286+00:00 (in 29.999588 seconds)
buildlogsarchiver stdout | 2025-02-07 05:12:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:03 UTC)" (scheduled at 2025-02-07 05:12:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:12:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 12, 33, 919935), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:12:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:12:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:12:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:03 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:56.713547+00:00 (in 18.997210 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:37 UTC)" (scheduled at 2025-02-07 05:12:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:12:38,174 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:12:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:48.732147+00:00 (in 9.999513 seconds)
notificationworker stdout | 2025-02-07 05:12:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:48 UTC)" (scheduled at 2025-02-07 05:12:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:12:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:12:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 38, 733004), True, datetime.datetime(2025, 2, 7, 5, 12, 38, 733004), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:12:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:12:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:12:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:12:39,188 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:12:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:12:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:54.423055+00:00 (in 13.998355 seconds)
gcworker stdout | 2025-02-07 05:12:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:10 UTC)" (scheduled at 2025-02-07 05:12:40.424250+00:00)
gcworker stdout | 2025-02-07 05:12:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:12:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904860431, None, 1, 0])
gcworker stdout | 2025-02-07 05:12:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:12:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:12:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:12:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:11.831521+00:00 (in 29.999485 seconds)
autopruneworker stdout | 2025-02-07 05:12:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:11 UTC)" (scheduled at 2025-02-07 05:12:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:12:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901561836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:12:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:12:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:12:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:12:43,140 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:12:43,678 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:12:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:12:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:59.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 05:12:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:59 UTC)" (scheduled at 2025-02-07 05:12:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:12:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:12:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:12:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:12:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:12:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:12:44,235 [235] [DEBUG] [app] Starting request: urn:request:48577332-30b5-43ba-a237-0103e6226586 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:44,235 [238] [DEBUG] [app] Starting request: urn:request:3460e2eb-7148-4aab-a9f5-108db5e66c65 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:12:44,238 [239] [DEBUG] [app] Starting request: urn:request:31e7551b-733c-4967-afe7-6ededc34c3e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:44,238 [246] [DEBUG] [app] Starting request: urn:request:a8a0271c-618f-4c57-870d-0eb4b3f6dd07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:44,239 [239] [DEBUG] [app] Ending request: urn:request:31e7551b-733c-4967-afe7-6ededc34c3e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31e7551b-733c-4967-afe7-6ededc34c3e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:12:44,239 [246] [DEBUG] [app] Ending request: urn:request:a8a0271c-618f-4c57-870d-0eb4b3f6dd07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a8a0271c-618f-4c57-870d-0eb4b3f6dd07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:12:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:12:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:12:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:44,242 [237] [DEBUG] [app] Starting request: urn:request:e48f6dd8-6c35-4879-b796-770a4afb5b1c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:44,242 [237] [DEBUG] [app] Ending request: urn:request:e48f6dd8-6c35-4879-b796-770a4afb5b1c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e48f6dd8-6c35-4879-b796-770a4afb5b1c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:12:44,242 [235] [DEBUG] [app] Starting request: urn:request:fffee81d-fd8c-4258-985e-1473c7bfa0d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:44,242 [235] [DEBUG] [app] Ending request: urn:request:fffee81d-fd8c-4258-985e-1473c7bfa0d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fffee81d-fd8c-4258-985e-1473c7bfa0d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:12:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:12:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:44,254 [235] [DEBUG] [app] Ending request: urn:request:48577332-30b5-43ba-a237-0103e6226586 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:48577332-30b5-43ba-a237-0103e6226586', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:12:44,254 [238] [DEBUG] [app] Ending request: urn:request:3460e2eb-7148-4aab-a9f5-108db5e66c65 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3460e2eb-7148-4aab-a9f5-108db5e66c65', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:12:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 05:12:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
logrotateworker stdout | 2025-02-07 05:12:44,274 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:12:45,662 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:12:46,485 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:12:46,526 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:12:47,478 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:12:47,481 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:12:47,485 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:12:47,488 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:12:47,490 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:12:47,791 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:48,097 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:12:48,502 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:12:48,505 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:12:48,508 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:12:48,588 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:12:48,639 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:12:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:50.814677+00:00 (in 2.082061 seconds)
notificationworker stdout | 2025-02-07 05:12:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:58 UTC)" (scheduled at 2025-02-07 05:12:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:12:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:12:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 48, 732881), True, datetime.datetime(2025, 2, 7, 5, 12, 48, 732881), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:12:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:12:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:12:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:12:58 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:12:48,847 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:12:49,211 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:12:49,619 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:12:49,782 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:12:49,851 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:12:49,854 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:12:49,857 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:12:49,862 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:12:49,865 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:12:49,868 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:12:49,870 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:12:49,874 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:12:49,876 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:12:50,042 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:12:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:12:58.732147+00:00 (in 7.917032 seconds)
notificationworker stdout | 2025-02-07 05:12:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:50 UTC)" (scheduled at 2025-02-07 05:12:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:12:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:12:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:12:51,472 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:12:51,677 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:12:52,036 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:12:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:12:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:10.424250+00:00 (in 16.000739 seconds)
gcworker stdout | 2025-02-07 05:12:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:24 UTC)" (scheduled at 2025-02-07 05:12:54.423055+00:00)
gcworker stdout | 2025-02-07 05:12:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:12:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:12:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:12:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:56.118855+00:00 (in 59.999491 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:12:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:56 UTC)" (scheduled at 2025-02-07 05:12:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:12:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:12:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:12:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:37.715878+00:00 (in 41.001873 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:56 UTC)" (scheduled at 2025-02-07 05:12:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:12:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:12:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:12:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:08.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 05:12:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:08 UTC)" (scheduled at 2025-02-07 05:12:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:12:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:12:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 12, 58, 732832), True, datetime.datetime(2025, 2, 7, 5, 12, 58, 732832), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:12:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:12:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:12:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:12:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:12:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:14.225378+00:00 (in 14.999527 seconds)
quotatotalworker stdout | 2025-02-07 05:12:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:14 UTC)" (scheduled at 2025-02-07 05:12:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:12:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:12:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:12:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:12:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:12:59,236 [235] [DEBUG] [app] Starting request: urn:request:0efc8115-c392-4450-b1bd-b37bf361c6ab (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:59,236 [237] [DEBUG] [app] Starting request: urn:request:b6306cc4-d9fa-4788-ad09-2ba8c148572c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:12:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:59,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:12:59,240 [239] [DEBUG] [app] Starting request: urn:request:8c54ef37-c4d6-40c5-bd65-91ca52ad3bd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:59,240 [245] [DEBUG] [app] Starting request: urn:request:6b1a6d97-160b-4a2e-979e-617a7b507826 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:12:59,240 [245] [DEBUG] [app] Ending request: urn:request:6b1a6d97-160b-4a2e-979e-617a7b507826 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b1a6d97-160b-4a2e-979e-617a7b507826', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:12:59,240 [239] [DEBUG] [app] Ending request: urn:request:8c54ef37-c4d6-40c5-bd65-91ca52ad3bd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c54ef37-c4d6-40c5-bd65-91ca52ad3bd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:12:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:12:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:12:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:59,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:12:59,243 [237] [DEBUG] [app] Starting request: urn:request:21e39fd7-9ba0-4854-b058-ca15bb742857 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:59,244 [237] [DEBUG] [app] Ending request: urn:request:21e39fd7-9ba0-4854-b058-ca15bb742857 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21e39fd7-9ba0-4854-b058-ca15bb742857', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:12:59,244 [236] [DEBUG] [app] Starting request: urn:request:faa1cc02-dc9c-4217-970f-99aa1835c623 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:12:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:12:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:59,244 [236] [DEBUG] [app] Ending request: urn:request:faa1cc02-dc9c-4217-970f-99aa1835c623 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:faa1cc02-dc9c-4217-970f-99aa1835c623', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:12:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:12:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:12:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:12:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:12:59,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:12:59,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:12:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:12:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:12:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:12:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:59,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:12:59,255 [237] [DEBUG] [app] Ending request: urn:request:b6306cc4-d9fa-4788-ad09-2ba8c148572c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b6306cc4-d9fa-4788-ad09-2ba8c148572c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:12:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:12:59,256 [235] [DEBUG] [app] Ending request: urn:request:0efc8115-c392-4450-b1bd-b37bf361c6ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0efc8115-c392-4450-b1bd-b37bf361c6ab', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:12:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:12:59,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:12:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:12:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
exportactionlogsworker stdout | 2025-02-07 05:12:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:12:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:04.936688+00:00 (in 4.998320 seconds)
exportactionlogsworker stdout | 2025-02-07 05:12:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:59 UTC)" (scheduled at 2025-02-07 05:12:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:12:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:12:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:13:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:13:00,951 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:13:02,748 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:13:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:13:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:23.420886+00:00 (in 20.001343 seconds)
repositorygcworker stdout | 2025-02-07 05:13:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:03 UTC)" (scheduled at 2025-02-07 05:13:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:13:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:13:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 3, 419812), True, datetime.datetime(2025, 2, 7, 5, 13, 3, 419812), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:13:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:13:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:13:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:13:03,901 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
buildlogsarchiver stdout | 2025-02-07 05:13:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:13:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:33.919286+00:00 (in 29.999583 seconds)
buildlogsarchiver stdout | 2025-02-07 05:13:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:33 UTC)" (scheduled at 2025-02-07 05:13:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:13:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 13, 3, 919943), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:13:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:13:03,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:13:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:13:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:13:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:59.937899+00:00 (in 55.000745 seconds)
exportactionlogsworker stdout | 2025-02-07 05:13:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:04 UTC)" (scheduled at 2025-02-07 05:13:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:13:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:13:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 4, 937503), True, datetime.datetime(2025, 2, 7, 5, 13, 4, 937503), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:13:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:13:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:13:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:13:08,210 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:13:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:13:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:18.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 05:13:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:18 UTC)" (scheduled at 2025-02-07 05:13:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:13:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:13:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 8, 732868), True, datetime.datetime(2025, 2, 7, 5, 13, 8, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:13:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:13:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:13:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:13:09,218 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:13:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:13:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:24.423055+00:00 (in 13.998344 seconds)
gcworker stdout | 2025-02-07 05:13:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:40 UTC)" (scheduled at 2025-02-07 05:13:10.424250+00:00)
gcworker stdout | 2025-02-07 05:13:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:13:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904890431, None, 1, 0])
gcworker stdout | 2025-02-07 05:13:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:13:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:13:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:13:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:41.831521+00:00 (in 29.999506 seconds)
autopruneworker stdout | 2025-02-07 05:13:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:41 UTC)" (scheduled at 2025-02-07 05:13:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:13:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901591836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:13:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:13:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:13:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:13:13,170 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:13:13,711 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:13:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:13:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:29.225378+00:00 (in 14.999527 seconds)
quotatotalworker stdout | 2025-02-07 05:13:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:29 UTC)" (scheduled at 2025-02-07 05:13:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:13:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:13:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:13:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:13:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:13:14,235 [235] [DEBUG] [app] Starting request: urn:request:ec008f92-67d6-4232-a577-48d413ab0f38 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:14,235 [238] [DEBUG] [app] Starting request: urn:request:96dcaeea-512b-46fc-a519-8c2f8c467a0d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:13:14,238 [246] [DEBUG] [app] Starting request: urn:request:846b2699-db96-4c25-81dd-2243ab49641f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:13:14,238 [246] [DEBUG] [app] Ending request: urn:request:846b2699-db96-4c25-81dd-2243ab49641f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:846b2699-db96-4c25-81dd-2243ab49641f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:13:14,239 [239] [DEBUG] [app] Starting request: urn:request:2ddcc585-f7f7-410f-b566-bf77c021a85c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:13:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:13:14,239 [239] [DEBUG] [app] Ending request: urn:request:2ddcc585-f7f7-410f-b566-bf77c021a85c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ddcc585-f7f7-410f-b566-bf77c021a85c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:13:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:13:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:13:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:14,241 [236] [DEBUG] [app] Starting request: urn:request:320f69e9-1aeb-4070-be65-ca28b74f30d0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:14,242 [236] [DEBUG] [app] Ending request: urn:request:320f69e9-1aeb-4070-be65-ca28b74f30d0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:320f69e9-1aeb-4070-be65-ca28b74f30d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:13:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:13:14,242 [238] [DEBUG] [app] Starting request: urn:request:20d8b14c-4405-4500-adc6-abbab61256e6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:14,243 [238] [DEBUG] [app] Ending request: urn:request:20d8b14c-4405-4500-adc6-abbab61256e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:20d8b14c-4405-4500-adc6-abbab61256e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:13:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:13:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:14,254 [238] [DEBUG] [app] Ending request: urn:request:96dcaeea-512b-46fc-a519-8c2f8c467a0d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:96dcaeea-512b-46fc-a519-8c2f8c467a0d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:13:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:13:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:13:14,254 [235] [DEBUG] [app] Ending request: urn:request:ec008f92-67d6-4232-a577-48d413ab0f38 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec008f92-67d6-4232-a577-48d413ab0f38', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:13:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:13:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
logrotateworker stdout | 2025-02-07 05:13:14,310 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:13:15,698 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:13:16,521 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:13:16,562 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:13:17,486 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:13:17,489 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:13:17,492 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:13:17,495 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:13:17,498 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:13:17,824 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:18,133 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:13:18,509 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:13:18,512 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:13:18,514 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:13:18,625 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:13:18,673 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:13:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:13:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:28.732147+00:00 (in 9.999594 seconds)
notificationworker stdout | 2025-02-07 05:13:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:28 UTC)" (scheduled at 2025-02-07 05:13:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:13:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:13:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 18, 732852), True, datetime.datetime(2025, 2, 7, 5, 13, 18, 732852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:13:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:13:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:13:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:28 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:13:18,884 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:13:19,247 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:13:19,637 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:13:19,818 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:13:19,860 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:13:19,867 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:13:19,871 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:13:19,874 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:13:19,877 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:13:19,880 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:13:19,883 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:13:19,887 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:13:19,890 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:13:20,078 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:13:21,509 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:13:21,713 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:13:22,070 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:13:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:13:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:32.420707+00:00 (in 8.999350 seconds)
repositorygcworker stdout | 2025-02-07 05:13:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:23 UTC)" (scheduled at 2025-02-07 05:13:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:13:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:13:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:13:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:13:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:40.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 05:13:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:54 UTC)" (scheduled at 2025-02-07 05:13:24.423055+00:00)
gcworker stdout | 2025-02-07 05:13:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:13:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:13:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:13:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:13:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:38.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 05:13:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:38 UTC)" (scheduled at 2025-02-07 05:13:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:13:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:13:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 28, 732854), True, datetime.datetime(2025, 2, 7, 5, 13, 28, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:13:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:13:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:13:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:13:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:13:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:44.225378+00:00 (in 14.999533 seconds)
quotatotalworker stdout | 2025-02-07 05:13:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:44 UTC)" (scheduled at 2025-02-07 05:13:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:13:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:13:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:13:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:13:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:13:29,235 [238] [DEBUG] [app] Starting request: urn:request:3b63b6c6-3d74-46a2-9eb0-4b3fcec92773 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:29,235 [237] [DEBUG] [app] Starting request: urn:request:134d1f0e-3f27-4281-8328-2ef63aa5aade (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:13:29,238 [245] [DEBUG] [app] Starting request: urn:request:60d50221-1324-45bb-9b83-663946755888 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:13:29,238 [246] [DEBUG] [app] Starting request: urn:request:c21ba1c2-2eea-49f1-ad97-29b65ac32543 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:13:29,238 [246] [DEBUG] [app] Ending request: urn:request:c21ba1c2-2eea-49f1-ad97-29b65ac32543 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c21ba1c2-2eea-49f1-ad97-29b65ac32543', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:13:29,238 [245] [DEBUG] [app] Ending request: urn:request:60d50221-1324-45bb-9b83-663946755888 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60d50221-1324-45bb-9b83-663946755888', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:13:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:13:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:13:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:29,241 [238] [DEBUG] [app] Starting request: urn:request:b020cbeb-a7c8-4d99-91bd-464afd3fdd47 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:29,241 [238] [DEBUG] [app] Ending request: urn:request:b020cbeb-a7c8-4d99-91bd-464afd3fdd47 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b020cbeb-a7c8-4d99-91bd-464afd3fdd47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:13:29,241 [236] [DEBUG] [app] Starting request: urn:request:14ed0f68-90b3-4522-9e54-1b611801e165 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:13:29,242 [236] [DEBUG] [app] Ending request: urn:request:14ed0f68-90b3-4522-9e54-1b611801e165 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:14ed0f68-90b3-4522-9e54-1b611801e165', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:13:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:13:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:13:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:29,253 [238] [DEBUG] [app] Ending request: urn:request:3b63b6c6-3d74-46a2-9eb0-4b3fcec92773 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3b63b6c6-3d74-46a2-9eb0-4b3fcec92773', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:13:29,253 [237] [DEBUG] [app] Ending request: urn:request:134d1f0e-3f27-4281-8328-2ef63aa5aade (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:134d1f0e-3f27-4281-8328-2ef63aa5aade', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:13:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:13:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:13:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:13:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 05:13:30,987 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:13:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:13:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:32.625945+00:00 (in 1.004369 seconds)
namespacegcworker stdout | 2025-02-07 05:13:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:31 UTC)" (scheduled at 2025-02-07 05:13:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:13:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:13:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 31, 621846), True, datetime.datetime(2025, 2, 7, 5, 13, 31, 621846), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:13:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:13:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:13:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:13:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:13:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:03.419110+00:00 (in 30.997930 seconds)
repositorygcworker stdout | 2025-02-07 05:13:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:18:32 UTC)" (scheduled at 2025-02-07 05:13:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:13:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 13, 32, 421466), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:13:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 13, 32, 421466), True, datetime.datetime(2025, 2, 7, 5, 13, 32, 421466), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:13:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 13, 32, 421466), True, datetime.datetime(2025, 2, 7, 5, 13, 32, 421466), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 13, 32, 421466), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:13:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:13:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:18:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:13:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:13:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:31.621131+00:00 (in 58.994724 seconds)
namespacegcworker stdout | 2025-02-07 05:13:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:32 UTC)" (scheduled at 2025-02-07 05:13:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:13:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:13:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:13:32,784 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:13:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:13:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:03.919286+00:00 (in 29.999565 seconds)
buildlogsarchiver stdout | 2025-02-07 05:13:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:03 UTC)" (scheduled at 2025-02-07 05:13:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:13:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 13, 33, 919970), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:13:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:13:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:13:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:13:33,937 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:56.713547+00:00 (in 18.997208 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:37 UTC)" (scheduled at 2025-02-07 05:13:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:13:38,230 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:13:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:13:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:48.732147+00:00 (in 9.999571 seconds)
notificationworker stdout | 2025-02-07 05:13:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:48 UTC)" (scheduled at 2025-02-07 05:13:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:13:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:13:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 38, 732843), True, datetime.datetime(2025, 2, 7, 5, 13, 38, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:13:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:13:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:13:38,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:13:39,234 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:13:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:13:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:54.423055+00:00 (in 13.998386 seconds)
gcworker stdout | 2025-02-07 05:13:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:10 UTC)" (scheduled at 2025-02-07 05:13:40.424250+00:00)
gcworker stdout | 2025-02-07 05:13:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:13:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904920431, None, 1, 0])
gcworker stdout | 2025-02-07 05:13:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:13:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:13:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:13:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:11.831521+00:00 (in 29.999551 seconds)
autopruneworker stdout | 2025-02-07 05:13:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:11 UTC)" (scheduled at 2025-02-07 05:13:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:13:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901621836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:13:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:13:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:13:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:13:43,186 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:13:43,734 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:13:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:13:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:59.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 05:13:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:59 UTC)" (scheduled at 2025-02-07 05:13:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:13:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:13:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:13:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:13:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:13:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:13:44,235 [236] [DEBUG] [app] Starting request: urn:request:2c1f26bb-f1dc-41f7-9559-fde3a7175382 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:44,235 [235] [DEBUG] [app] Starting request: urn:request:d1a59cc4-494f-437e-80fc-e3aecaa82af1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:13:44,239 [239] [DEBUG] [app] Starting request: urn:request:34921200-fd11-4604-9aad-ba4431dfc4f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:13:44,239 [246] [DEBUG] [app] Starting request: urn:request:c289874d-15bc-46ae-992a-2b3a9efc9260 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:13:44,239 [239] [DEBUG] [app] Ending request: urn:request:34921200-fd11-4604-9aad-ba4431dfc4f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34921200-fd11-4604-9aad-ba4431dfc4f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:13:44,239 [246] [DEBUG] [app] Ending request: urn:request:c289874d-15bc-46ae-992a-2b3a9efc9260 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c289874d-15bc-46ae-992a-2b3a9efc9260', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:13:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:13:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:13:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:44,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:44,242 [238] [DEBUG] [app] Starting request: urn:request:c69c512a-6fa5-4c5d-971f-a113fcd7787c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:44,242 [238] [DEBUG] [app] Ending request: urn:request:c69c512a-6fa5-4c5d-971f-a113fcd7787c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c69c512a-6fa5-4c5d-971f-a113fcd7787c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:13:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:13:44,243 [235] [DEBUG] [app] Starting request: urn:request:f3d1f7d8-a9df-4772-8ffa-6ae79c4c8324 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:44,243 [235] [DEBUG] [app] Ending request: urn:request:f3d1f7d8-a9df-4772-8ffa-6ae79c4c8324 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3d1f7d8-a9df-4772-8ffa-6ae79c4c8324', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:13:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:13:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:44,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:44,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:44,255 [235] [DEBUG] [app] Ending request: urn:request:d1a59cc4-494f-437e-80fc-e3aecaa82af1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1a59cc4-494f-437e-80fc-e3aecaa82af1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:13:44,255 [236] [DEBUG] [app] Ending request: urn:request:2c1f26bb-f1dc-41f7-9559-fde3a7175382 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2c1f26bb-f1dc-41f7-9559-fde3a7175382', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:13:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:13:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:13:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:13:44,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
logrotateworker stdout | 2025-02-07 05:13:44,322 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:13:45,705 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:13:46,549 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:13:46,598 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:13:47,495 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:13:47,498 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:13:47,502 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:13:47,504 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:13:47,507 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:13:47,855 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:48,141 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:13:48,518 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:13:48,520 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:13:48,523 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:13:48,634 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:13:48,680 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:13:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:13:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:50.814677+00:00 (in 2.082118 seconds)
notificationworker stdout | 2025-02-07 05:13:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:58 UTC)" (scheduled at 2025-02-07 05:13:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:13:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:13:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 48, 732829), True, datetime.datetime(2025, 2, 7, 5, 13, 48, 732829), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:13:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:13:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:13:48,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:13:58 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:13:48,917 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:13:49,284 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:13:49,673 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:13:49,842 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:13:49,868 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:13:49,878 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:13:49,882 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:13:49,886 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:13:49,888 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:13:49,891 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:13:49,895 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:13:49,898 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:13:49,900 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:13:50,099 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:13:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:13:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:13:58.732147+00:00 (in 7.917008 seconds)
notificationworker stdout | 2025-02-07 05:13:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:50 UTC)" (scheduled at 2025-02-07 05:13:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:13:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:13:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:13:51,528 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:13:51,749 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:13:52,077 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:13:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:13:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:10.424250+00:00 (in 16.000744 seconds)
gcworker stdout | 2025-02-07 05:13:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:24 UTC)" (scheduled at 2025-02-07 05:13:54.423055+00:00)
gcworker stdout | 2025-02-07 05:13:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:13:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:13:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:13:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:56.118855+00:00 (in 59.999471 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:13:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:56 UTC)" (scheduled at 2025-02-07 05:13:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:13:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:13:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:13:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:37.715878+00:00 (in 41.001892 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:56 UTC)" (scheduled at 2025-02-07 05:13:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:13:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:13:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:13:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:08.732147+00:00 (in 9.999530 seconds)
notificationworker stdout | 2025-02-07 05:13:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:08 UTC)" (scheduled at 2025-02-07 05:13:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:13:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:13:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 13, 58, 732928), True, datetime.datetime(2025, 2, 7, 5, 13, 58, 732928), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:13:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:13:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:13:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:13:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:13:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:14.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 05:13:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:14 UTC)" (scheduled at 2025-02-07 05:13:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:13:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:13:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:13:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:13:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:13:59,235 [235] [DEBUG] [app] Starting request: urn:request:35a13cfe-c1e8-4fc0-9e31-e69bc1eff257 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:59,235 [238] [DEBUG] [app] Starting request: urn:request:4f55c27d-e6bf-424d-92cd-3dd15c8a5044 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:13:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:13:59,238 [239] [DEBUG] [app] Starting request: urn:request:a2e5c9b2-c600-42c2-9477-6d46676efcd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:13:59,238 [245] [DEBUG] [app] Starting request: urn:request:68203c24-6b3e-499d-8f01-d6aaa7a3ddce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:13:59,238 [239] [DEBUG] [app] Ending request: urn:request:a2e5c9b2-c600-42c2-9477-6d46676efcd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2e5c9b2-c600-42c2-9477-6d46676efcd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:13:59,238 [245] [DEBUG] [app] Ending request: urn:request:68203c24-6b3e-499d-8f01-d6aaa7a3ddce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68203c24-6b3e-499d-8f01-d6aaa7a3ddce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:13:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:13:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:13:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:13:59,241 [237] [DEBUG] [app] Starting request: urn:request:5cb397bf-9358-4980-b86d-2778a2a64196 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:59,241 [236] [DEBUG] [app] Starting request: urn:request:c315008d-27b7-4a17-9d5a-951868b94d92 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:13:59,242 [237] [DEBUG] [app] Ending request: urn:request:5cb397bf-9358-4980-b86d-2778a2a64196 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5cb397bf-9358-4980-b86d-2778a2a64196', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:13:59,242 [236] [DEBUG] [app] Ending request: urn:request:c315008d-27b7-4a17-9d5a-951868b94d92 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c315008d-27b7-4a17-9d5a-951868b94d92', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:13:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:13:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:13:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:13:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:13:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:13:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:13:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:13:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:13:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:13:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:13:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:13:59,253 [238] [DEBUG] [app] Ending request: urn:request:4f55c27d-e6bf-424d-92cd-3dd15c8a5044 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f55c27d-e6bf-424d-92cd-3dd15c8a5044', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:13:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:13:59,253 [235] [DEBUG] [app] Ending request: urn:request:35a13cfe-c1e8-4fc0-9e31-e69bc1eff257 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:35a13cfe-c1e8-4fc0-9e31-e69bc1eff257', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 05:13:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:13:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:13:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:13:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:13:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:13:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:13:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:04.936688+00:00 (in 4.998298 seconds)
exportactionlogsworker stdout | 2025-02-07 05:13:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:59 UTC)" (scheduled at 2025-02-07 05:13:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:13:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:13:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:14:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:14:01,010 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:14:02,820 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:14:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:14:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:23.420886+00:00 (in 20.001313 seconds)
repositorygcworker stdout | 2025-02-07 05:14:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:03 UTC)" (scheduled at 2025-02-07 05:14:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:14:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:14:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 3, 419851), True, datetime.datetime(2025, 2, 7, 5, 14, 3, 419851), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:14:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:14:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:14:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:14:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:14:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:33.919286+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2025-02-07 05:14:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:33 UTC)" (scheduled at 2025-02-07 05:14:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:14:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 14, 3, 920019), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:14:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:14:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:14:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:14:03,973 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:14:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:14:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:52.937648+00:00 (in 48.000498 seconds)
exportactionlogsworker stdout | 2025-02-07 05:14:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:04 UTC)" (scheduled at 2025-02-07 05:14:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:14:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:14:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 4, 937461), True, datetime.datetime(2025, 2, 7, 5, 14, 4, 937461), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:14:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:14:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:14:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:14:08,266 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:14:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:14:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:18.732147+00:00 (in 9.999614 seconds)
notificationworker stdout | 2025-02-07 05:14:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:18 UTC)" (scheduled at 2025-02-07 05:14:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:14:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:14:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 8, 732843), True, datetime.datetime(2025, 2, 7, 5, 14, 8, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:14:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:14:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:14:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:14:09,245 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:14:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:14:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:24.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 05:14:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:40 UTC)" (scheduled at 2025-02-07 05:14:10.424250+00:00)
gcworker stdout | 2025-02-07 05:14:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:14:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904950430, None, 1, 0])
gcworker stdout | 2025-02-07 05:14:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:14:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:14:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:14:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:41.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 05:14:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:41 UTC)" (scheduled at 2025-02-07 05:14:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:14:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901651836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:14:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:14:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:14:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:14:13,207 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:14:13,752 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:14:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:14:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:29.225378+00:00 (in 14.999553 seconds)
quotatotalworker stdout | 2025-02-07 05:14:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:29 UTC)" (scheduled at 2025-02-07 05:14:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:14:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:14:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:14:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:14:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:14:14,235 [235] [DEBUG] [app] Starting request: urn:request:a9ffe666-86df-4f9f-8c3a-c72c0ef50c39 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:14,235 [238] [DEBUG] [app] Starting request: urn:request:5119e9b3-13ac-4258-b51f-8ace2ef89a4f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:14:14,238 [239] [DEBUG] [app] Starting request: urn:request:3372402a-aac8-4235-99f7-34dfa3d743e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:14,238 [246] [DEBUG] [app] Starting request: urn:request:0924b536-5e3e-4e8b-9ecf-2a66bde42580 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:14,238 [239] [DEBUG] [app] Ending request: urn:request:3372402a-aac8-4235-99f7-34dfa3d743e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3372402a-aac8-4235-99f7-34dfa3d743e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:14,238 [246] [DEBUG] [app] Ending request: urn:request:0924b536-5e3e-4e8b-9ecf-2a66bde42580 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0924b536-5e3e-4e8b-9ecf-2a66bde42580', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:14:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:14:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:14,241 [235] [DEBUG] [app] Starting request: urn:request:5f9bd316-fc8d-4e51-a681-c5996c046466 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:14,241 [237] [DEBUG] [app] Starting request: urn:request:30f32299-b363-4de0-970f-6bd205995358 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:14,241 [235] [DEBUG] [app] Ending request: urn:request:5f9bd316-fc8d-4e51-a681-c5996c046466 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f9bd316-fc8d-4e51-a681-c5996c046466', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:14:14,241 [237] [DEBUG] [app] Ending request: urn:request:30f32299-b363-4de0-970f-6bd205995358 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:30f32299-b363-4de0-970f-6bd205995358', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:14:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:14:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:14:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:14,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:14,252 [238] [DEBUG] [app] Ending request: urn:request:5119e9b3-13ac-4258-b51f-8ace2ef89a4f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5119e9b3-13ac-4258-b51f-8ace2ef89a4f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:14,252 [235] [DEBUG] [app] Ending request: urn:request:a9ffe666-86df-4f9f-8c3a-c72c0ef50c39 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a9ffe666-86df-4f9f-8c3a-c72c0ef50c39', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:14,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:14:14,252 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:14:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:14:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
logrotateworker stdout | 2025-02-07 05:14:14,358 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:14:15,741 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:14:16,570 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:14:16,635 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:14:17,504 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:14:17,508 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:14:17,512 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:14:17,515 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:14:17,517 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:14:17,867 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:18,177 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:14:18,526 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:14:18,529 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:14:18,532 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:14:18,650 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:14:18,711 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:14:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:14:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:28.732147+00:00 (in 9.999533 seconds)
notificationworker stdout | 2025-02-07 05:14:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:28 UTC)" (scheduled at 2025-02-07 05:14:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:14:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:14:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 18, 732918), True, datetime.datetime(2025, 2, 7, 5, 14, 18, 732918), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:14:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:14:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:14:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:28 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:14:18,954 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:14:19,320 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:14:19,709 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:14:19,850 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:14:19,880 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:14:19,890 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:14:19,893 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:14:19,896 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:14:19,899 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:14:19,902 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:14:19,905 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:14:19,909 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:14:19,912 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:14:20,135 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:14:21,534 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:14:21,764 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:14:22,095 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:14:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:14:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:03.419110+00:00 (in 39.997743 seconds)
repositorygcworker stdout | 2025-02-07 05:14:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:23 UTC)" (scheduled at 2025-02-07 05:14:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:14:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:14:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:14:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:14:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:40.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 05:14:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:54 UTC)" (scheduled at 2025-02-07 05:14:24.423055+00:00)
gcworker stdout | 2025-02-07 05:14:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:14:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:14:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:14:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:14:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:38.732147+00:00 (in 9.999547 seconds)
notificationworker stdout | 2025-02-07 05:14:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:38 UTC)" (scheduled at 2025-02-07 05:14:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:14:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:14:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 28, 732875), True, datetime.datetime(2025, 2, 7, 5, 14, 28, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:14:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:14:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:14:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:14:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:14:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:44.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 05:14:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:44 UTC)" (scheduled at 2025-02-07 05:14:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:14:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:14:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:14:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:14:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:14:29,235 [235] [DEBUG] [app] Starting request: urn:request:bef7f9a6-9edb-4803-992d-985877d44aa0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:29,235 [236] [DEBUG] [app] Starting request: urn:request:3ea903d0-d1d0-4816-9110-897cf32d589c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:14:29,238 [246] [DEBUG] [app] Starting request: urn:request:dccae483-93bd-4585-acd5-199c560a942a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:29,238 [245] [DEBUG] [app] Starting request: urn:request:84e97be9-0d49-4862-b91b-8a7d3f3261ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:29,239 [246] [DEBUG] [app] Ending request: urn:request:dccae483-93bd-4585-acd5-199c560a942a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dccae483-93bd-4585-acd5-199c560a942a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:29,239 [245] [DEBUG] [app] Ending request: urn:request:84e97be9-0d49-4862-b91b-8a7d3f3261ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:84e97be9-0d49-4862-b91b-8a7d3f3261ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:14:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:14:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:29,243 [237] [DEBUG] [app] Starting request: urn:request:6645e476-11bf-4c48-982f-5a009c5d29b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:29,243 [238] [DEBUG] [app] Starting request: urn:request:eb520f60-4b91-48a9-acab-13b8bc589b81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:29,243 [237] [DEBUG] [app] Ending request: urn:request:6645e476-11bf-4c48-982f-5a009c5d29b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6645e476-11bf-4c48-982f-5a009c5d29b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:14:29,243 [238] [DEBUG] [app] Ending request: urn:request:eb520f60-4b91-48a9-acab-13b8bc589b81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb520f60-4b91-48a9-acab-13b8bc589b81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:14:29,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:14:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:14:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:29,255 [236] [DEBUG] [app] Ending request: urn:request:3ea903d0-d1d0-4816-9110-897cf32d589c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ea903d0-d1d0-4816-9110-897cf32d589c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:14:29,255 [235] [DEBUG] [app] Ending request: urn:request:bef7f9a6-9edb-4803-992d-985877d44aa0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bef7f9a6-9edb-4803-992d-985877d44aa0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:14:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:14:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 05:14:31,033 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:14:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:14:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:32.625945+00:00 (in 1.004362 seconds)
namespacegcworker stdout | 2025-02-07 05:14:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:31 UTC)" (scheduled at 2025-02-07 05:14:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:14:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:14:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 31, 621853), True, datetime.datetime(2025, 2, 7, 5, 14, 31, 621853), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:14:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:14:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:14:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:14:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:14:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:31.621131+00:00 (in 58.994719 seconds)
namespacegcworker stdout | 2025-02-07 05:14:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:32 UTC)" (scheduled at 2025-02-07 05:14:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:14:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:14:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:14:32,852 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:14:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:14:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:03.919286+00:00 (in 29.999563 seconds)
buildlogsarchiver stdout | 2025-02-07 05:14:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:03 UTC)" (scheduled at 2025-02-07 05:14:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:14:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 14, 33, 919913), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:14:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:14:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:14:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:14:34,009 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:56.713547+00:00 (in 18.997153 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:37 UTC)" (scheduled at 2025-02-07 05:14:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:14:38,277 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:14:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:14:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:48.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 05:14:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:48 UTC)" (scheduled at 2025-02-07 05:14:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:14:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:14:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 38, 732891), True, datetime.datetime(2025, 2, 7, 5, 14, 38, 732891), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:14:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:14:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:14:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:14:39,281 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:14:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:14:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:54.423055+00:00 (in 13.998369 seconds)
gcworker stdout | 2025-02-07 05:14:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:10 UTC)" (scheduled at 2025-02-07 05:14:40.424250+00:00)
gcworker stdout | 2025-02-07 05:14:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:14:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738904980431, None, 1, 0])
gcworker stdout | 2025-02-07 05:14:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:14:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:14:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:14:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:11.831521+00:00 (in 29.999498 seconds)
autopruneworker stdout | 2025-02-07 05:14:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:11 UTC)" (scheduled at 2025-02-07 05:14:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:14:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901681836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:14:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:14:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:14:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:14:43,243 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:14:43,788 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:14:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:14:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:59.225378+00:00 (in 14.999520 seconds)
quotatotalworker stdout | 2025-02-07 05:14:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:59 UTC)" (scheduled at 2025-02-07 05:14:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:14:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:14:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:14:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:14:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:14:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:14:44,236 [235] [DEBUG] [app] Starting request: urn:request:0529feb0-d5bb-422b-a6ad-8c5d76d1861c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:44,236 [236] [DEBUG] [app] Starting request: urn:request:2b0f66f5-9580-495c-acc2-074a373c7639 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:14:44,239 [244] [DEBUG] [app] Starting request: urn:request:c237e65c-3c51-402f-8491-be4c3e017832 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:44,239 [246] [DEBUG] [app] Starting request: urn:request:e3a65c7e-2205-4252-a31f-d5a9e7309873 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:44,239 [244] [DEBUG] [app] Ending request: urn:request:c237e65c-3c51-402f-8491-be4c3e017832 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c237e65c-3c51-402f-8491-be4c3e017832', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:44,239 [246] [DEBUG] [app] Ending request: urn:request:e3a65c7e-2205-4252-a31f-d5a9e7309873 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e3a65c7e-2205-4252-a31f-d5a9e7309873', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:14:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:14:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:14:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:44,242 [236] [DEBUG] [app] Starting request: urn:request:273e6351-5202-4b7c-8029-e4edfa32e6d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:44,243 [236] [DEBUG] [app] Ending request: urn:request:273e6351-5202-4b7c-8029-e4edfa32e6d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:273e6351-5202-4b7c-8029-e4edfa32e6d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:14:44,242 [238] [DEBUG] [app] Starting request: urn:request:5794c86c-40af-4ab8-a184-9ec8e0303419 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:44,243 [238] [DEBUG] [app] Ending request: urn:request:5794c86c-40af-4ab8-a184-9ec8e0303419 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5794c86c-40af-4ab8-a184-9ec8e0303419', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:14:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:14:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:14:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:44,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:44,255 [235] [DEBUG] [app] Ending request: urn:request:0529feb0-d5bb-422b-a6ad-8c5d76d1861c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0529feb0-d5bb-422b-a6ad-8c5d76d1861c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:14:44,255 [236] [DEBUG] [app] Ending request: urn:request:2b0f66f5-9580-495c-acc2-074a373c7639 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b0f66f5-9580-495c-acc2-074a373c7639', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:14:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:14:44,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
logrotateworker stdout | 2025-02-07 05:14:44,394 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:14:45,763 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:14:46,605 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:14:46,658 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:14:47,516 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:14:47,520 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:14:47,522 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:14:47,525 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:14:47,528 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:14:47,887 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:48,207 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:14:48,535 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:14:48,538 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:14:48,541 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:14:48,675 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
notificationworker stdout | 2025-02-07 05:14:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:14:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:50.814677+00:00 (in 2.082105 seconds)
notificationworker stdout | 2025-02-07 05:14:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:58 UTC)" (scheduled at 2025-02-07 05:14:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:14:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:14:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 48, 732864), True, datetime.datetime(2025, 2, 7, 5, 14, 48, 732864), 0, 'notification/%', 50, 1, 0])
servicekey stdout | 2025-02-07 05:14:48,734 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:14:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:14:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:14:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:14:58 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:14:48,979 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:14:49,350 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:14:49,745 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:14:49,886 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:14:49,891 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:14:49,899 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:14:49,903 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:14:49,906 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:14:49,909 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:14:49,912 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:14:49,916 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:14:49,919 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:14:49,921 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:14:50,168 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:14:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:14:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:58.732147+00:00 (in 7.917033 seconds)
notificationworker stdout | 2025-02-07 05:14:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:50 UTC)" (scheduled at 2025-02-07 05:14:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:14:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:14:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:14:51,570 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:14:51,784 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:14:52,132 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
exportactionlogsworker stdout | 2025-02-07 05:14:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:14:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:14:59.937899+00:00 (in 6.999812 seconds)
exportactionlogsworker stdout | 2025-02-07 05:14:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:19:52 UTC)" (scheduled at 2025-02-07 05:14:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:14:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 14, 52, 938349), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:14:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 14, 52, 938349), True, datetime.datetime(2025, 2, 7, 5, 14, 52, 938349), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:14:52,946 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 14, 52, 938349), True, datetime.datetime(2025, 2, 7, 5, 14, 52, 938349), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 14, 52, 938349), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:14:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:14:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:19:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:14:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:14:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:10.424250+00:00 (in 16.000741 seconds)
gcworker stdout | 2025-02-07 05:14:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:24 UTC)" (scheduled at 2025-02-07 05:14:54.423055+00:00)
gcworker stdout | 2025-02-07 05:14:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:14:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:14:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:14:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:56.118855+00:00 (in 59.999503 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:14:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:56 UTC)" (scheduled at 2025-02-07 05:14:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:14:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:14:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:14:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:37.715878+00:00 (in 41.001791 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:56 UTC)" (scheduled at 2025-02-07 05:14:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:14:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:14:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:14:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:08.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 05:14:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:08 UTC)" (scheduled at 2025-02-07 05:14:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:14:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:14:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 14, 58, 732908), True, datetime.datetime(2025, 2, 7, 5, 14, 58, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:14:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:14:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:14:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:14:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:14:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:14.225378+00:00 (in 14.999539 seconds)
quotatotalworker stdout | 2025-02-07 05:14:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:14 UTC)" (scheduled at 2025-02-07 05:14:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:14:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:14:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:14:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:14:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:14:59,235 [236] [DEBUG] [app] Starting request: urn:request:47b1c7f6-d452-44ef-b514-3d53c4c7728d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:59,235 [235] [DEBUG] [app] Starting request: urn:request:0a9aa042-2199-42d1-922c-3c5db5cccce3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:14:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:14:59,238 [246] [DEBUG] [app] Starting request: urn:request:502b11a6-e839-4a4f-ab8c-5a6743b01f10 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:59,238 [239] [DEBUG] [app] Starting request: urn:request:40796942-b337-4a0b-826e-0e511b865de4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:14:59,238 [246] [DEBUG] [app] Ending request: urn:request:502b11a6-e839-4a4f-ab8c-5a6743b01f10 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:502b11a6-e839-4a4f-ab8c-5a6743b01f10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:59,239 [239] [DEBUG] [app] Ending request: urn:request:40796942-b337-4a0b-826e-0e511b865de4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40796942-b337-4a0b-826e-0e511b865de4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:14:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:14:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:14:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:14:59,242 [236] [DEBUG] [app] Starting request: urn:request:8573ebfc-d82c-4bee-8c5f-800a395b96a7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:59,242 [236] [DEBUG] [app] Ending request: urn:request:8573ebfc-d82c-4bee-8c5f-800a395b96a7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8573ebfc-d82c-4bee-8c5f-800a395b96a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:14:59,242 [238] [DEBUG] [app] Starting request: urn:request:01745bdd-76c0-4f8d-a1a1-97f2fba63fb3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:14:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:14:59,242 [238] [DEBUG] [app] Ending request: urn:request:01745bdd-76c0-4f8d-a1a1-97f2fba63fb3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01745bdd-76c0-4f8d-a1a1-97f2fba63fb3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:14:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:14:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:14:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:14:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:14:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:14:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:14:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:14:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:14:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:14:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:14:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:14:59,254 [236] [DEBUG] [app] Ending request: urn:request:47b1c7f6-d452-44ef-b514-3d53c4c7728d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47b1c7f6-d452-44ef-b514-3d53c4c7728d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:59,254 [235] [DEBUG] [app] Ending request: urn:request:0a9aa042-2199-42d1-922c-3c5db5cccce3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0a9aa042-2199-42d1-922c-3c5db5cccce3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:14:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:14:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:14:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:14:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:14:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:14:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:14:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:14:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:04.936688+00:00 (in 4.998307 seconds)
exportactionlogsworker stdout | 2025-02-07 05:14:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:59 UTC)" (scheduled at 2025-02-07 05:14:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:14:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:14:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:15:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:15:01,070 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:15:02,888 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:15:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:15:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:23.420886+00:00 (in 20.001303 seconds)
repositorygcworker stdout | 2025-02-07 05:15:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:03 UTC)" (scheduled at 2025-02-07 05:15:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:15:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:15:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 3, 419873), True, datetime.datetime(2025, 2, 7, 5, 15, 3, 419873), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:15:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:15:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:15:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:15:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:15:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:33.919286+00:00 (in 29.999516 seconds)
buildlogsarchiver stdout | 2025-02-07 05:15:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:33 UTC)" (scheduled at 2025-02-07 05:15:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:15:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 15, 3, 920052), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:15:03,928 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:15:03,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:15:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:15:04,045 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:15:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:15:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:59.937899+00:00 (in 55.000731 seconds)
exportactionlogsworker stdout | 2025-02-07 05:15:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:04 UTC)" (scheduled at 2025-02-07 05:15:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:15:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:15:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 4, 937514), True, datetime.datetime(2025, 2, 7, 5, 15, 4, 937514), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:15:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:15:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:15:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:15:08,304 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:15:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:15:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:18.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:15:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:18 UTC)" (scheduled at 2025-02-07 05:15:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:15:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:15:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 8, 732876), True, datetime.datetime(2025, 2, 7, 5, 15, 8, 732876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:15:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:15:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:15:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:15:09,310 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:15:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:15:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:24.423055+00:00 (in 13.998386 seconds)
gcworker stdout | 2025-02-07 05:15:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:40 UTC)" (scheduled at 2025-02-07 05:15:10.424250+00:00)
gcworker stdout | 2025-02-07 05:15:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:15:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905010431, None, 1, 0])
gcworker stdout | 2025-02-07 05:15:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:15:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:15:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:15:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:41.831521+00:00 (in 29.999515 seconds)
autopruneworker stdout | 2025-02-07 05:15:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:41 UTC)" (scheduled at 2025-02-07 05:15:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:15:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901711836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:15:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:15:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:15:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:15:13,250 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:15:13,798 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:15:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:15:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:29.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 05:15:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:29 UTC)" (scheduled at 2025-02-07 05:15:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:15:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:15:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:15:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:15:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:15:14,236 [235] [DEBUG] [app] Starting request: urn:request:4ee8b829-ad53-4611-b7f1-f26424c2e521 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:14,236 [236] [DEBUG] [app] Starting request: urn:request:3286d92a-2b74-431e-9106-0c0a6c97f4c7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:15:14,239 [239] [DEBUG] [app] Starting request: urn:request:36c88da5-f404-4748-b722-bd6418065d17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:14,239 [245] [DEBUG] [app] Starting request: urn:request:f52e3922-cc02-4e07-9ee3-4092cc4d339a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:14,240 [245] [DEBUG] [app] Ending request: urn:request:f52e3922-cc02-4e07-9ee3-4092cc4d339a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f52e3922-cc02-4e07-9ee3-4092cc4d339a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:15:14,240 [239] [DEBUG] [app] Ending request: urn:request:36c88da5-f404-4748-b722-bd6418065d17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36c88da5-f404-4748-b722-bd6418065d17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:15:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:15:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:15:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:14,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:14,243 [237] [DEBUG] [app] Starting request: urn:request:ac4744f8-7be2-47d6-9287-9c368a4089ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:15:14,243 [238] [DEBUG] [app] Starting request: urn:request:25c81b65-715d-4596-98a6-fb3b6cc399ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:15:14,244 [237] [DEBUG] [app] Ending request: urn:request:ac4744f8-7be2-47d6-9287-9c368a4089ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ac4744f8-7be2-47d6-9287-9c368a4089ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:15:14,244 [238] [DEBUG] [app] Ending request: urn:request:25c81b65-715d-4596-98a6-fb3b6cc399ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25c81b65-715d-4596-98a6-fb3b6cc399ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:15:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:15:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:15:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:14,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:14,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:14,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:14,249 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:14,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:14,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:14,255 [235] [DEBUG] [app] Ending request: urn:request:4ee8b829-ad53-4611-b7f1-f26424c2e521 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ee8b829-ad53-4611-b7f1-f26424c2e521', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:14,255 [236] [DEBUG] [app] Ending request: urn:request:3286d92a-2b74-431e-9106-0c0a6c97f4c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3286d92a-2b74-431e-9106-0c0a6c97f4c7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:15:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:15:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:15:14,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
logrotateworker stdout | 2025-02-07 05:15:14,406 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:15:15,796 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:15:16,618 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:15:16,694 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:15:17,525 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:15:17,528 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:15:17,531 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:15:17,534 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:15:17,537 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:15:17,907 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:18,243 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:15:18,543 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:15:18,547 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:15:18,549 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:15:18,711 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
notificationworker stdout | 2025-02-07 05:15:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:15:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:28.732147+00:00 (in 9.999575 seconds)
notificationworker stdout | 2025-02-07 05:15:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:28 UTC)" (scheduled at 2025-02-07 05:15:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:15:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:15:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 18, 732842), True, datetime.datetime(2025, 2, 7, 5, 15, 18, 732842), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:15:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:15:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:15:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:28 UTC)" executed successfully
servicekey stdout | 2025-02-07 05:15:18,770 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:15:19,012 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:15:19,386 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:15:19,774 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:15:19,902 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:15:19,908 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:15:19,911 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:15:19,914 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:15:19,918 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:15:19,922 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
securityscanningnotificationworker stdout | 2025-02-07 05:15:19,924 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
gunicorn-registry stdout | 2025-02-07 05:15:19,927 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:15:19,930 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:15:19,932 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
manifestbackfillworker stdout | 2025-02-07 05:15:20,205 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:15:21,588 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:15:21,809 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:15:22,151 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:15:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:15:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:03.419110+00:00 (in 39.997753 seconds)
repositorygcworker stdout | 2025-02-07 05:15:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:23 UTC)" (scheduled at 2025-02-07 05:15:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:15:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:15:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:15:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:15:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:40.424250+00:00 (in 16.000735 seconds)
gcworker stdout | 2025-02-07 05:15:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:54 UTC)" (scheduled at 2025-02-07 05:15:24.423055+00:00)
gcworker stdout | 2025-02-07 05:15:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:15:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:15:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:15:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:15:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:38.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 05:15:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:38 UTC)" (scheduled at 2025-02-07 05:15:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:15:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:15:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 28, 732858), True, datetime.datetime(2025, 2, 7, 5, 15, 28, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:15:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:15:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:15:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:15:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:15:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:44.225378+00:00 (in 14.999568 seconds)
quotatotalworker stdout | 2025-02-07 05:15:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:44 UTC)" (scheduled at 2025-02-07 05:15:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:15:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:15:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:15:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:15:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:15:29,235 [237] [DEBUG] [app] Starting request: urn:request:01c05e72-02f2-48ce-9391-f578799e6759 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:29,235 [235] [DEBUG] [app] Starting request: urn:request:38f154c5-e9a2-4bc1-9395-f74187ce2532 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:15:29,239 [239] [DEBUG] [app] Starting request: urn:request:bbaae0ca-8401-4f52-afc6-5e0271d78178 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:29,239 [245] [DEBUG] [app] Starting request: urn:request:49ed38d0-a167-4718-91e3-ea7c9e0e07c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:29,239 [245] [DEBUG] [app] Ending request: urn:request:49ed38d0-a167-4718-91e3-ea7c9e0e07c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:49ed38d0-a167-4718-91e3-ea7c9e0e07c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:15:29,239 [239] [DEBUG] [app] Ending request: urn:request:bbaae0ca-8401-4f52-afc6-5e0271d78178 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bbaae0ca-8401-4f52-afc6-5e0271d78178', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:15:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:15:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:15:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:29,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:29,243 [238] [DEBUG] [app] Starting request: urn:request:5dbf3f82-c48d-452d-8db4-526606c9f156 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:15:29,243 [236] [DEBUG] [app] Starting request: urn:request:f6493207-03c7-4c07-af43-4b7b6a0ee2b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:15:29,244 [238] [DEBUG] [app] Ending request: urn:request:5dbf3f82-c48d-452d-8db4-526606c9f156 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5dbf3f82-c48d-452d-8db4-526606c9f156', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:15:29,244 [236] [DEBUG] [app] Ending request: urn:request:f6493207-03c7-4c07-af43-4b7b6a0ee2b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6493207-03c7-4c07-af43-4b7b6a0ee2b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:15:29,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:15:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:15:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:29,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:29,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:29,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:29,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:29,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:29,255 [235] [DEBUG] [app] Ending request: urn:request:38f154c5-e9a2-4bc1-9395-f74187ce2532 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:38f154c5-e9a2-4bc1-9395-f74187ce2532', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:29,255 [237] [DEBUG] [app] Ending request: urn:request:01c05e72-02f2-48ce-9391-f578799e6759 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01c05e72-02f2-48ce-9391-f578799e6759', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:15:29,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:15:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:15:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
storagereplication stdout | 2025-02-07 05:15:31,107 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:15:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:15:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:32.625945+00:00 (in 1.004350 seconds)
namespacegcworker stdout | 2025-02-07 05:15:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:31 UTC)" (scheduled at 2025-02-07 05:15:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:15:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:15:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 31, 621914), True, datetime.datetime(2025, 2, 7, 5, 15, 31, 621914), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:15:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:15:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:15:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:15:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:15:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:31.621131+00:00 (in 58.994730 seconds)
namespacegcworker stdout | 2025-02-07 05:15:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:32 UTC)" (scheduled at 2025-02-07 05:15:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:15:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:15:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:15:32,919 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:15:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:15:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:03.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 05:15:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:03 UTC)" (scheduled at 2025-02-07 05:15:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:15:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 15, 33, 919993), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:15:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:15:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:15:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:15:34,082 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:56.713547+00:00 (in 18.997163 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:37 UTC)" (scheduled at 2025-02-07 05:15:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:15:38,330 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:15:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:15:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:48.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 05:15:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:48 UTC)" (scheduled at 2025-02-07 05:15:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:15:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:15:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 38, 732850), True, datetime.datetime(2025, 2, 7, 5, 15, 38, 732850), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:15:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:15:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:15:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:15:39,341 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:15:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:15:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:54.423055+00:00 (in 13.998352 seconds)
gcworker stdout | 2025-02-07 05:15:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:10 UTC)" (scheduled at 2025-02-07 05:15:40.424250+00:00)
gcworker stdout | 2025-02-07 05:15:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:15:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905040431, None, 1, 0])
gcworker stdout | 2025-02-07 05:15:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:15:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:15:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:15:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:11.831521+00:00 (in 29.999580 seconds)
autopruneworker stdout | 2025-02-07 05:15:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:11 UTC)" (scheduled at 2025-02-07 05:15:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:15:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901741836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:15:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:15:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:15:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:15:43,274 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:15:43,831 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:15:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:15:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:59.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 05:15:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:59 UTC)" (scheduled at 2025-02-07 05:15:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:15:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:15:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:15:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:15:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:15:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:15:44,235 [238] [DEBUG] [app] Starting request: urn:request:d6c0a68e-c5fe-4cb1-a274-083c8f082c97 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:44,235 [235] [DEBUG] [app] Starting request: urn:request:6bab4f77-b3ce-4e10-ad4a-3d667160d75d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:15:44,238 [246] [DEBUG] [app] Starting request: urn:request:dbd7b290-f6df-46ec-bc6d-59a06631cf6f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:44,238 [239] [DEBUG] [app] Starting request: urn:request:114251b7-4cde-44c5-9558-ec8d47e4b7d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:44,238 [246] [DEBUG] [app] Ending request: urn:request:dbd7b290-f6df-46ec-bc6d-59a06631cf6f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dbd7b290-f6df-46ec-bc6d-59a06631cf6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:15:44,239 [239] [DEBUG] [app] Ending request: urn:request:114251b7-4cde-44c5-9558-ec8d47e4b7d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:114251b7-4cde-44c5-9558-ec8d47e4b7d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:15:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:15:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:15:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:15:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:44,242 [235] [DEBUG] [app] Starting request: urn:request:e1949f1d-bd5c-4489-994c-f1ca047e198e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:15:44,242 [235] [DEBUG] [app] Ending request: urn:request:e1949f1d-bd5c-4489-994c-f1ca047e198e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e1949f1d-bd5c-4489-994c-f1ca047e198e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:15:44,242 [236] [DEBUG] [app] Starting request: urn:request:528a215a-4d2b-4817-ab04-5dd118248316 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:15:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:15:44,242 [236] [DEBUG] [app] Ending request: urn:request:528a215a-4d2b-4817-ab04-5dd118248316 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:528a215a-4d2b-4817-ab04-5dd118248316', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:15:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:15:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:44,244 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 5, 15, 44, 243419), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 5, 15, 44, 243436), 'quay', None, datetime.datetime(2025, 2, 6, 5, 15, 44, 243446), datetime.datetime(2025, 1, 31, 5, 15, 44, 243455), None])
gunicorn-web stdout | 2025-02-07 05:15:44,244 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 5, 15, 44, 243627), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 5, 15, 44, 243644), 'quay', None, datetime.datetime(2025, 2, 6, 5, 15, 44, 243655), datetime.datetime(2025, 1, 31, 5, 15, 44, 243663), None])
gunicorn-web stdout | 2025-02-07 05:15:44,250 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:44,250 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:44,250 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:44,251 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:44,254 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:44,254 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:44,255 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:44,256 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:44,257 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:44,257 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:44,258 [235] [DEBUG] [app] Ending request: urn:request:6bab4f77-b3ce-4e10-ad4a-3d667160d75d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6bab4f77-b3ce-4e10-ad4a-3d667160d75d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:44,258 [238] [DEBUG] [app] Ending request: urn:request:d6c0a68e-c5fe-4cb1-a274-083c8f082c97 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d6c0a68e-c5fe-4cb1-a274-083c8f082c97', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:44,258 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:15:44,258 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:15:44,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:15:44,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.024)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.025)
logrotateworker stdout | 2025-02-07 05:15:44,436 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:15:45,832 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:15:46,629 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:15:46,707 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:15:47,532 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:15:47,536 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:15:47,539 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:15:47,541 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:15:47,544 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:15:47,943 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:48,269 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:15:48,553 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:15:48,556 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:15:48,559 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
globalpromstats stdout | 2025-02-07 05:15:48,730 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
notificationworker stdout | 2025-02-07 05:15:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:15:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:50.814677+00:00 (in 2.082121 seconds)
notificationworker stdout | 2025-02-07 05:15:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:58 UTC)" (scheduled at 2025-02-07 05:15:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:15:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:15:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 48, 732883), True, datetime.datetime(2025, 2, 7, 5, 15, 48, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:15:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:15:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:15:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:15:58 UTC)" executed successfully
servicekey stdout | 2025-02-07 05:15:48,806 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:15:49,049 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:15:49,423 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:15:49,811 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:15:49,914 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:15:49,919 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:15:49,922 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:15:49,924 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:15:49,927 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:15:49,929 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:15:49,934 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:15:49,937 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:15:49,941 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:15:49,960 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:15:50,241 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:15:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:15:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:15:58.732147+00:00 (in 7.917032 seconds)
notificationworker stdout | 2025-02-07 05:15:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:50 UTC)" (scheduled at 2025-02-07 05:15:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:15:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:15:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:15:51,618 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:15:51,818 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:15:52,187 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:15:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:15:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:10.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 05:15:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:24 UTC)" (scheduled at 2025-02-07 05:15:54.423055+00:00)
gcworker stdout | 2025-02-07 05:15:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:15:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:15:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:15:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:56.118855+00:00 (in 59.999508 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:15:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:56 UTC)" (scheduled at 2025-02-07 05:15:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:15:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:15:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:15:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:37.715878+00:00 (in 41.001839 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:56 UTC)" (scheduled at 2025-02-07 05:15:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:15:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:15:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:15:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:08.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 05:15:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:08 UTC)" (scheduled at 2025-02-07 05:15:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:15:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:15:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 15, 58, 732863), True, datetime.datetime(2025, 2, 7, 5, 15, 58, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:15:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:15:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:15:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:15:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:15:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:14.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 05:15:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:14 UTC)" (scheduled at 2025-02-07 05:15:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:15:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:15:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:15:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:15:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:15:59,235 [237] [DEBUG] [app] Starting request: urn:request:a3c21052-6927-4855-8e49-4ace2c29001a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:59,235 [238] [DEBUG] [app] Starting request: urn:request:b2aa447f-5677-43eb-a9e5-abf7de17c3b1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:15:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:15:59,238 [245] [DEBUG] [app] Starting request: urn:request:bf671e83-b993-4b4e-8dfa-3b49a96a592e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:59,238 [239] [DEBUG] [app] Starting request: urn:request:acf8dadb-c4ca-4475-8c29-dfed8f7f0e6d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:15:59,238 [245] [DEBUG] [app] Ending request: urn:request:bf671e83-b993-4b4e-8dfa-3b49a96a592e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bf671e83-b993-4b4e-8dfa-3b49a96a592e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:15:59,238 [239] [DEBUG] [app] Ending request: urn:request:acf8dadb-c4ca-4475-8c29-dfed8f7f0e6d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:acf8dadb-c4ca-4475-8c29-dfed8f7f0e6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:15:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:15:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:15:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:15:59,242 [236] [DEBUG] [app] Starting request: urn:request:ddedd61e-15e4-4bbb-8f9b-c01b5b920cba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:15:59,242 [235] [DEBUG] [app] Starting request: urn:request:1a79989c-afa5-4fdc-be1e-308e0bbad444 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:15:59,242 [236] [DEBUG] [app] Ending request: urn:request:ddedd61e-15e4-4bbb-8f9b-c01b5b920cba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ddedd61e-15e4-4bbb-8f9b-c01b5b920cba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:15:59,242 [235] [DEBUG] [app] Ending request: urn:request:1a79989c-afa5-4fdc-be1e-308e0bbad444 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1a79989c-afa5-4fdc-be1e-308e0bbad444', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:15:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:15:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:15:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:15:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:15:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:15:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:15:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:59,244 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 5, 15, 59, 243902), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 5, 15, 59, 243927), 'quay', None, datetime.datetime(2025, 2, 6, 5, 15, 59, 243940), datetime.datetime(2025, 1, 31, 5, 15, 59, 243953), None])
gunicorn-web stdout | 2025-02-07 05:15:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:59,250 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:15:59,250 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:15:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:59,253 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:15:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:15:59,253 [238] [DEBUG] [app] Ending request: urn:request:b2aa447f-5677-43eb-a9e5-abf7de17c3b1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2aa447f-5677-43eb-a9e5-abf7de17c3b1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:15:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:15:59,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:15:59,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:15:59,256 [237] [DEBUG] [app] Ending request: urn:request:a3c21052-6927-4855-8e49-4ace2c29001a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a3c21052-6927-4855-8e49-4ace2c29001a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:15:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:15:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.023)
gunicorn-web stdout | 2025-02-07 05:15:59,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:15:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:15:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:15:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:04.936688+00:00 (in 4.998284 seconds)
exportactionlogsworker stdout | 2025-02-07 05:15:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:59 UTC)" (scheduled at 2025-02-07 05:15:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:15:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:15:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:16:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:16:01,129 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:16:02,944 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:16:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:16:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:23.420886+00:00 (in 20.001332 seconds)
repositorygcworker stdout | 2025-02-07 05:16:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:03 UTC)" (scheduled at 2025-02-07 05:16:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:16:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:16:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 3, 419827), True, datetime.datetime(2025, 2, 7, 5, 16, 3, 419827), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:16:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:16:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:16:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:16:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:16:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:33.919286+00:00 (in 29.999541 seconds)
buildlogsarchiver stdout | 2025-02-07 05:16:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:33 UTC)" (scheduled at 2025-02-07 05:16:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:16:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 16, 3, 920029), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:16:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:16:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:16:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:16:04,089 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:16:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:16:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:59.937899+00:00 (in 55.000783 seconds)
exportactionlogsworker stdout | 2025-02-07 05:16:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:04 UTC)" (scheduled at 2025-02-07 05:16:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:16:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:16:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 4, 937409), True, datetime.datetime(2025, 2, 7, 5, 16, 4, 937409), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:16:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:16:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:16:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:16:08,366 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:16:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:16:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:18.732147+00:00 (in 9.999400 seconds)
notificationworker stdout | 2025-02-07 05:16:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:18 UTC)" (scheduled at 2025-02-07 05:16:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:16:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:16:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 8, 733136), True, datetime.datetime(2025, 2, 7, 5, 16, 8, 733136), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:16:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:16:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:16:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:16:09,367 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:16:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:16:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:24.423055+00:00 (in 13.998375 seconds)
gcworker stdout | 2025-02-07 05:16:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:40 UTC)" (scheduled at 2025-02-07 05:16:10.424250+00:00)
gcworker stdout | 2025-02-07 05:16:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:16:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905070431, None, 1, 0])
gcworker stdout | 2025-02-07 05:16:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:16:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:16:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:16:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:41.831521+00:00 (in 29.999517 seconds)
autopruneworker stdout | 2025-02-07 05:16:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:41 UTC)" (scheduled at 2025-02-07 05:16:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:16:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901771836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:16:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:16:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:16:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:16:13,311 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:16:13,841 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:16:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:16:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:29.225378+00:00 (in 14.999555 seconds)
quotatotalworker stdout | 2025-02-07 05:16:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:29 UTC)" (scheduled at 2025-02-07 05:16:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:16:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:16:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:16:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:16:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:16:14,236 [238] [DEBUG] [app] Starting request: urn:request:51951d53-9f35-4fa8-8b85-a042dc280e78 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:14,236 [237] [DEBUG] [app] Starting request: urn:request:a880df8a-521f-441e-88ea-6c8dc40c46cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:16:14,239 [239] [DEBUG] [app] Starting request: urn:request:a3391e94-3689-4f98-9e0f-76dded939cc3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:14,239 [246] [DEBUG] [app] Starting request: urn:request:a37dfaf8-4728-44b1-a469-d0804557da3e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:14,240 [246] [DEBUG] [app] Ending request: urn:request:a37dfaf8-4728-44b1-a469-d0804557da3e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a37dfaf8-4728-44b1-a469-d0804557da3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:16:14,240 [239] [DEBUG] [app] Ending request: urn:request:a3391e94-3689-4f98-9e0f-76dded939cc3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3391e94-3689-4f98-9e0f-76dded939cc3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:16:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:16:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:16:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:14,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:14,243 [236] [DEBUG] [app] Starting request: urn:request:a65a4bc6-4348-47e6-b525-7781a9c39e78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:14,243 [235] [DEBUG] [app] Starting request: urn:request:54c2c4dd-0cb9-4211-9ea4-2d2d917fd548 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:14,244 [236] [DEBUG] [app] Ending request: urn:request:a65a4bc6-4348-47e6-b525-7781a9c39e78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a65a4bc6-4348-47e6-b525-7781a9c39e78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:16:14,244 [235] [DEBUG] [app] Ending request: urn:request:54c2c4dd-0cb9-4211-9ea4-2d2d917fd548 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:54c2c4dd-0cb9-4211-9ea4-2d2d917fd548', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:16:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:14,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:14,255 [237] [DEBUG] [app] Ending request: urn:request:a880df8a-521f-441e-88ea-6c8dc40c46cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a880df8a-521f-441e-88ea-6c8dc40c46cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:14,255 [238] [DEBUG] [app] Ending request: urn:request:51951d53-9f35-4fa8-8b85-a042dc280e78 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:51951d53-9f35-4fa8-8b85-a042dc280e78', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:16:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
logrotateworker stdout | 2025-02-07 05:16:14,458 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:16:15,854 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:16:16,644 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:16:16,731 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:16:17,541 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:16:17,545 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:16:17,549 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:16:17,552 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:16:17,554 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:16:17,979 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:18,305 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:16:18,562 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:16:18,565 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:16:18,567 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:16:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:16:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:28.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 05:16:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:28 UTC)" (scheduled at 2025-02-07 05:16:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:16:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:16:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 18, 732936), True, datetime.datetime(2025, 2, 7, 5, 16, 18, 732936), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:16:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:16:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:16:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:16:18,748 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:16:18,828 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:16:19,079 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:16:19,450 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:16:19,819 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:16:19,922 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:16:19,927 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:16:19,930 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:16:19,934 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:16:19,938 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:16:19,941 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:16:19,943 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:16:19,948 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:16:19,951 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:16:19,992 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:16:20,277 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:16:21,647 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:16:21,853 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:16:22,223 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:16:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:16:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:03.419110+00:00 (in 39.997735 seconds)
repositorygcworker stdout | 2025-02-07 05:16:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:23 UTC)" (scheduled at 2025-02-07 05:16:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:16:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:16:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:16:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:16:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:40.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 05:16:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:54 UTC)" (scheduled at 2025-02-07 05:16:24.423055+00:00)
gcworker stdout | 2025-02-07 05:16:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:16:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:16:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:16:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:16:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:38.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 05:16:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:38 UTC)" (scheduled at 2025-02-07 05:16:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:16:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:16:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 28, 732861), True, datetime.datetime(2025, 2, 7, 5, 16, 28, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:16:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:16:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:16:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:16:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:16:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:44.225378+00:00 (in 14.999532 seconds)
quotatotalworker stdout | 2025-02-07 05:16:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:44 UTC)" (scheduled at 2025-02-07 05:16:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:16:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:16:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:16:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:16:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:16:29,235 [237] [DEBUG] [app] Starting request: urn:request:62dcbdfc-c5b4-4f0a-9562-35a03b7ee404 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:29,235 [235] [DEBUG] [app] Starting request: urn:request:fe4282d5-b7a8-4f66-9e22-4bf78cc746b0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:16:29,238 [239] [DEBUG] [app] Starting request: urn:request:484e41d7-f704-4703-8415-0a3267f2bb0a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:29,238 [246] [DEBUG] [app] Starting request: urn:request:ff624b65-533b-4ae4-8660-56d9842826f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:29,239 [239] [DEBUG] [app] Ending request: urn:request:484e41d7-f704-4703-8415-0a3267f2bb0a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:484e41d7-f704-4703-8415-0a3267f2bb0a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:16:29,239 [246] [DEBUG] [app] Ending request: urn:request:ff624b65-533b-4ae4-8660-56d9842826f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff624b65-533b-4ae4-8660-56d9842826f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:16:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:16:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:16:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:29,242 [235] [DEBUG] [app] Starting request: urn:request:74f84c7b-0e01-4195-a1a2-4d1fa23e45de (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:29,242 [236] [DEBUG] [app] Starting request: urn:request:7d83862d-e57e-451b-9b3c-947d0be46b77 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:29,242 [235] [DEBUG] [app] Ending request: urn:request:74f84c7b-0e01-4195-a1a2-4d1fa23e45de (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:74f84c7b-0e01-4195-a1a2-4d1fa23e45de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:16:29,242 [236] [DEBUG] [app] Ending request: urn:request:7d83862d-e57e-451b-9b3c-947d0be46b77 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d83862d-e57e-451b-9b3c-947d0be46b77', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:16:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:16:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:29,254 [237] [DEBUG] [app] Ending request: urn:request:62dcbdfc-c5b4-4f0a-9562-35a03b7ee404 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62dcbdfc-c5b4-4f0a-9562-35a03b7ee404', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:29,254 [235] [DEBUG] [app] Ending request: urn:request:fe4282d5-b7a8-4f66-9e22-4bf78cc746b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fe4282d5-b7a8-4f66-9e22-4bf78cc746b0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:16:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
storagereplication stdout | 2025-02-07 05:16:31,166 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:16:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:16:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:32.625945+00:00 (in 1.004367 seconds)
namespacegcworker stdout | 2025-02-07 05:16:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:31 UTC)" (scheduled at 2025-02-07 05:16:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:16:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:16:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 31, 621857), True, datetime.datetime(2025, 2, 7, 5, 16, 31, 621857), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:16:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:16:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:16:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:16:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:16:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:13.624301+00:00 (in 40.997866 seconds)
namespacegcworker stdout | 2025-02-07 05:16:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:32 UTC)" (scheduled at 2025-02-07 05:16:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:16:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:16:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:16:32,979 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:16:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:16:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:03.919286+00:00 (in 29.999513 seconds)
buildlogsarchiver stdout | 2025-02-07 05:16:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:03 UTC)" (scheduled at 2025-02-07 05:16:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:16:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 16, 33, 920015), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:16:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:16:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:16:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:16:34,118 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:56.713547+00:00 (in 18.997189 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:37 UTC)" (scheduled at 2025-02-07 05:16:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:16:38,402 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:16:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:16:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:48.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 05:16:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:48 UTC)" (scheduled at 2025-02-07 05:16:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:16:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:16:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 38, 732905), True, datetime.datetime(2025, 2, 7, 5, 16, 38, 732905), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:16:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:16:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:16:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:16:39,402 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:16:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:16:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:54.423055+00:00 (in 13.998222 seconds)
gcworker stdout | 2025-02-07 05:16:40,425 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:10 UTC)" (scheduled at 2025-02-07 05:16:40.424250+00:00)
gcworker stdout | 2025-02-07 05:16:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:16:40,432 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905100432, None, 1, 0])
gcworker stdout | 2025-02-07 05:16:40,434 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:16:40,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:16:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:16:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:11.831521+00:00 (in 29.999485 seconds)
autopruneworker stdout | 2025-02-07 05:16:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:11 UTC)" (scheduled at 2025-02-07 05:16:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:16:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901801836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:16:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:16:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:16:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:16:43,343 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:16:43,869 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:16:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:16:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:59.225378+00:00 (in 14.999534 seconds)
quotatotalworker stdout | 2025-02-07 05:16:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:59 UTC)" (scheduled at 2025-02-07 05:16:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:16:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:16:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:16:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:16:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:16:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:16:44,235 [237] [DEBUG] [app] Starting request: urn:request:43bb87c9-7987-454d-96d6-a49c96e02679 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:44,235 [238] [DEBUG] [app] Starting request: urn:request:61413277-d63f-4fa9-acb7-9281ad0cdb2a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:16:44,238 [239] [DEBUG] [app] Starting request: urn:request:0a397e4e-064e-42ce-9f0d-440ba88f9f2d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:44,238 [245] [DEBUG] [app] Starting request: urn:request:d752004f-c829-4a1c-9ed5-08d883d03c13 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:44,238 [245] [DEBUG] [app] Ending request: urn:request:d752004f-c829-4a1c-9ed5-08d883d03c13 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d752004f-c829-4a1c-9ed5-08d883d03c13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:16:44,238 [239] [DEBUG] [app] Ending request: urn:request:0a397e4e-064e-42ce-9f0d-440ba88f9f2d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0a397e4e-064e-42ce-9f0d-440ba88f9f2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:16:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:16:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:44,241 [237] [DEBUG] [app] Starting request: urn:request:8f6f72e0-4134-4ef3-bae7-6a0702434359 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:44,241 [236] [DEBUG] [app] Starting request: urn:request:7bce7022-01f3-4b3e-9b49-b38857c52e90 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:44,241 [237] [DEBUG] [app] Ending request: urn:request:8f6f72e0-4134-4ef3-bae7-6a0702434359 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8f6f72e0-4134-4ef3-bae7-6a0702434359', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:16:44,241 [236] [DEBUG] [app] Ending request: urn:request:7bce7022-01f3-4b3e-9b49-b38857c52e90 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7bce7022-01f3-4b3e-9b49-b38857c52e90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:16:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:16:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:16:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:44,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:44,253 [237] [DEBUG] [app] Ending request: urn:request:43bb87c9-7987-454d-96d6-a49c96e02679 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43bb87c9-7987-454d-96d6-a49c96e02679', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:44,253 [238] [DEBUG] [app] Ending request: urn:request:61413277-d63f-4fa9-acb7-9281ad0cdb2a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61413277-d63f-4fa9-acb7-9281ad0cdb2a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:16:44,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
logrotateworker stdout | 2025-02-07 05:16:44,495 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:16:45,888 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:16:46,680 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:16:46,758 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:16:47,549 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:16:47,552 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:16:47,555 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:16:47,558 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:16:47,561 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:16:48,012 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:48,328 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:16:48,569 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:16:48,572 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:16:48,576 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:16:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:16:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:50.814677+00:00 (in 2.082084 seconds)
notificationworker stdout | 2025-02-07 05:16:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:58 UTC)" (scheduled at 2025-02-07 05:16:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:16:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:16:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 48, 732904), True, datetime.datetime(2025, 2, 7, 5, 16, 48, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:16:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:16:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:16:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:16:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:16:48,758 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:16:48,841 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:16:49,111 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:16:49,487 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:16:49,855 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:16:49,932 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:16:49,935 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:16:49,938 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:16:49,945 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:16:49,948 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:16:49,953 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:16:49,956 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:16:49,959 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:16:49,961 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:16:50,029 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:16:50,289 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:16:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:16:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:16:58.732147+00:00 (in 7.917022 seconds)
notificationworker stdout | 2025-02-07 05:16:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:50 UTC)" (scheduled at 2025-02-07 05:16:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:16:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:16:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:16:51,671 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:16:51,889 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:16:52,258 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:16:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:16:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 05:16:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:24 UTC)" (scheduled at 2025-02-07 05:16:54.423055+00:00)
gcworker stdout | 2025-02-07 05:16:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:16:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:16:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:16:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:56.118855+00:00 (in 59.999530 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:16:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:56 UTC)" (scheduled at 2025-02-07 05:16:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:16:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:16:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:16:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:37.715878+00:00 (in 41.001858 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:56 UTC)" (scheduled at 2025-02-07 05:16:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:16:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:16:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:16:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:08.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 05:16:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:08 UTC)" (scheduled at 2025-02-07 05:16:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:16:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:16:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 16, 58, 732874), True, datetime.datetime(2025, 2, 7, 5, 16, 58, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:16:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:16:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:16:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:16:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:16:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:14.225378+00:00 (in 14.999533 seconds)
quotatotalworker stdout | 2025-02-07 05:16:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:14 UTC)" (scheduled at 2025-02-07 05:16:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:16:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:16:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:16:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:16:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:16:59,235 [235] [DEBUG] [app] Starting request: urn:request:a30ad3ea-25af-4e2f-a66f-70d997c58e98 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:59,235 [238] [DEBUG] [app] Starting request: urn:request:6b0beb89-fef5-49e3-8d55-e02eb303fe19 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:16:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:16:59,238 [239] [DEBUG] [app] Starting request: urn:request:6ef3ed1f-5211-466b-8f32-74b59924c44c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:59,238 [245] [DEBUG] [app] Starting request: urn:request:2a840374-96c3-4d63-a8cb-b92b7cf4e29d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:16:59,239 [239] [DEBUG] [app] Ending request: urn:request:6ef3ed1f-5211-466b-8f32-74b59924c44c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ef3ed1f-5211-466b-8f32-74b59924c44c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:16:59,239 [245] [DEBUG] [app] Ending request: urn:request:2a840374-96c3-4d63-a8cb-b92b7cf4e29d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a840374-96c3-4d63-a8cb-b92b7cf4e29d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:16:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:16:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:16:59,242 [235] [DEBUG] [app] Starting request: urn:request:c5deab51-447f-4cb1-bb33-54c7143450f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:59,242 [237] [DEBUG] [app] Starting request: urn:request:14ce6b79-beb2-4263-8e00-5bc1481e6ae0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:16:59,242 [235] [DEBUG] [app] Ending request: urn:request:c5deab51-447f-4cb1-bb33-54c7143450f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5deab51-447f-4cb1-bb33-54c7143450f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:16:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:16:59,242 [237] [DEBUG] [app] Ending request: urn:request:14ce6b79-beb2-4263-8e00-5bc1481e6ae0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:14ce6b79-beb2-4263-8e00-5bc1481e6ae0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:16:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:16:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:16:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:16:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:16:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:16:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:16:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:16:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:16:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:16:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:16:59,254 [235] [DEBUG] [app] Ending request: urn:request:a30ad3ea-25af-4e2f-a66f-70d997c58e98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a30ad3ea-25af-4e2f-a66f-70d997c58e98', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:59,255 [238] [DEBUG] [app] Ending request: urn:request:6b0beb89-fef5-49e3-8d55-e02eb303fe19 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b0beb89-fef5-49e3-8d55-e02eb303fe19', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:16:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:16:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:16:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:16:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:16:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:16:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:16:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:04.936688+00:00 (in 4.998311 seconds)
exportactionlogsworker stdout | 2025-02-07 05:16:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:59 UTC)" (scheduled at 2025-02-07 05:16:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:16:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:16:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:17:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:17:01,202 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:17:02,987 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:17:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:17:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:23.420886+00:00 (in 20.001336 seconds)
repositorygcworker stdout | 2025-02-07 05:17:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:03 UTC)" (scheduled at 2025-02-07 05:17:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:17:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:17:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 3, 419827), True, datetime.datetime(2025, 2, 7, 5, 17, 3, 419827), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:17:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:17:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:17:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:17:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:17:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:33.919286+00:00 (in 29.999577 seconds)
buildlogsarchiver stdout | 2025-02-07 05:17:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:33 UTC)" (scheduled at 2025-02-07 05:17:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:17:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 17, 3, 919887), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:17:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:17:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:17:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:17:04,154 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:17:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:17:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:59.937899+00:00 (in 55.000633 seconds)
exportactionlogsworker stdout | 2025-02-07 05:17:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:04 UTC)" (scheduled at 2025-02-07 05:17:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:17:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:17:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 4, 937595), True, datetime.datetime(2025, 2, 7, 5, 17, 4, 937595), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:17:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:17:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:17:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:17:08,430 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:17:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:18.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 05:17:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:18 UTC)" (scheduled at 2025-02-07 05:17:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:17:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:17:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 8, 732882), True, datetime.datetime(2025, 2, 7, 5, 17, 8, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:17:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:17:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:17:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:17:09,437 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:17:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:17:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:24.423055+00:00 (in 13.998356 seconds)
gcworker stdout | 2025-02-07 05:17:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:40 UTC)" (scheduled at 2025-02-07 05:17:10.424250+00:00)
gcworker stdout | 2025-02-07 05:17:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:17:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905130431, None, 1, 0])
gcworker stdout | 2025-02-07 05:17:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:17:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:17:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:17:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:41.831521+00:00 (in 29.999512 seconds)
autopruneworker stdout | 2025-02-07 05:17:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:41 UTC)" (scheduled at 2025-02-07 05:17:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:17:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901831836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:17:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:17:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:17:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:17:13,354 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
namespacegcworker stdout | 2025-02-07 05:17:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:17:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:31.621131+00:00 (in 17.996377 seconds)
namespacegcworker stdout | 2025-02-07 05:17:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:22:13 UTC)" (scheduled at 2025-02-07 05:17:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:17:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 17, 13, 625005), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:17:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 17, 13, 625005), True, datetime.datetime(2025, 2, 7, 5, 17, 13, 625005), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:17:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 17, 13, 625005), True, datetime.datetime(2025, 2, 7, 5, 17, 13, 625005), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 17, 13, 625005), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:17:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:17:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:22:13 UTC)" executed successfully
builder stdout | 2025-02-07 05:17:13,901 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:17:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:17:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:29.225378+00:00 (in 14.999547 seconds)
quotatotalworker stdout | 2025-02-07 05:17:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:29 UTC)" (scheduled at 2025-02-07 05:17:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:17:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:17:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:17:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:17:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:17:14,235 [237] [DEBUG] [app] Starting request: urn:request:9faa5cbb-7340-493e-8c35-20f3c60f9804 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:14,235 [238] [DEBUG] [app] Starting request: urn:request:729b286d-240f-441f-a597-97a976a5f602 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:17:14,238 [245] [DEBUG] [app] Starting request: urn:request:fcb2dd2b-f93b-4d4e-9da1-285b44c91647 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:14,238 [246] [DEBUG] [app] Starting request: urn:request:fd414705-9de3-4354-a62f-a5f8160fe327 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:14,238 [245] [DEBUG] [app] Ending request: urn:request:fcb2dd2b-f93b-4d4e-9da1-285b44c91647 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fcb2dd2b-f93b-4d4e-9da1-285b44c91647', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:17:14,238 [246] [DEBUG] [app] Ending request: urn:request:fd414705-9de3-4354-a62f-a5f8160fe327 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd414705-9de3-4354-a62f-a5f8160fe327', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:17:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:17:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:14,241 [236] [DEBUG] [app] Starting request: urn:request:48834b3b-d1d7-4202-b8ed-46da1f7839ab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:14,241 [237] [DEBUG] [app] Starting request: urn:request:17098e1d-0089-4183-85b1-4cdaa37b8e51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:14,242 [236] [DEBUG] [app] Ending request: urn:request:48834b3b-d1d7-4202-b8ed-46da1f7839ab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48834b3b-d1d7-4202-b8ed-46da1f7839ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:17:14,242 [237] [DEBUG] [app] Ending request: urn:request:17098e1d-0089-4183-85b1-4cdaa37b8e51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:17098e1d-0089-4183-85b1-4cdaa37b8e51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:17:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:14,253 [238] [DEBUG] [app] Ending request: urn:request:729b286d-240f-441f-a597-97a976a5f602 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:729b286d-240f-441f-a597-97a976a5f602', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:14,253 [237] [DEBUG] [app] Ending request: urn:request:9faa5cbb-7340-493e-8c35-20f3c60f9804 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9faa5cbb-7340-493e-8c35-20f3c60f9804', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:17:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
logrotateworker stdout | 2025-02-07 05:17:14,531 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:17:15,924 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:17:16,716 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:17:16,795 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:17:17,556 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:17:17,559 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:17:17,563 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:17:17,566 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:17:17,568 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:17:18,049 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:18,364 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:17:18,578 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:17:18,581 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:17:18,584 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:17:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:28.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 05:17:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:28 UTC)" (scheduled at 2025-02-07 05:17:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:17:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:17:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 18, 732858), True, datetime.datetime(2025, 2, 7, 5, 17, 18, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:17:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:17:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:17:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:17:18,790 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:17:18,877 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:17:19,129 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:17:19,499 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:17:19,871 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:17:19,941 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:17:19,944 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:17:19,948 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:17:19,955 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:17:19,958 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:17:19,963 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:17:19,966 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:17:19,969 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:17:19,971 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:17:20,065 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:17:20,318 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:17:21,690 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:17:21,926 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:17:22,279 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:17:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:17:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:03.419110+00:00 (in 39.997741 seconds)
repositorygcworker stdout | 2025-02-07 05:17:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:23 UTC)" (scheduled at 2025-02-07 05:17:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:17:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:17:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:17:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:17:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:40.424250+00:00 (in 16.000729 seconds)
gcworker stdout | 2025-02-07 05:17:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:54 UTC)" (scheduled at 2025-02-07 05:17:24.423055+00:00)
gcworker stdout | 2025-02-07 05:17:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:17:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:17:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:17:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:33.814166+00:00 (in 5.081552 seconds)
notificationworker stdout | 2025-02-07 05:17:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:38 UTC)" (scheduled at 2025-02-07 05:17:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:17:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:17:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 28, 732936), True, datetime.datetime(2025, 2, 7, 5, 17, 28, 732936), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:17:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:17:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:17:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:17:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:17:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:44.225378+00:00 (in 14.999534 seconds)
quotatotalworker stdout | 2025-02-07 05:17:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:44 UTC)" (scheduled at 2025-02-07 05:17:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:17:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:17:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:17:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:17:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:17:29,235 [235] [DEBUG] [app] Starting request: urn:request:7fcd115f-cbf3-4aab-9c99-055553e0828f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:29,235 [238] [DEBUG] [app] Starting request: urn:request:e99193cc-0a53-4620-8e02-52b48069e337 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:17:29,238 [239] [DEBUG] [app] Starting request: urn:request:b72f314b-813b-4e62-9667-9cda38b555c0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:29,239 [246] [DEBUG] [app] Starting request: urn:request:5111d360-379f-468f-9c30-b1677382842b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:29,239 [239] [DEBUG] [app] Ending request: urn:request:b72f314b-813b-4e62-9667-9cda38b555c0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b72f314b-813b-4e62-9667-9cda38b555c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:17:29,239 [246] [DEBUG] [app] Ending request: urn:request:5111d360-379f-468f-9c30-b1677382842b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5111d360-379f-468f-9c30-b1677382842b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:17:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:17:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:17:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:29,242 [238] [DEBUG] [app] Starting request: urn:request:b6bef1ae-03e1-48c2-b2d8-ed06e8352bcf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:29,242 [238] [DEBUG] [app] Ending request: urn:request:b6bef1ae-03e1-48c2-b2d8-ed06e8352bcf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b6bef1ae-03e1-48c2-b2d8-ed06e8352bcf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:17:29,242 [237] [DEBUG] [app] Starting request: urn:request:829da946-39a6-4ab2-bd6e-b8bd37c627f4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:29,243 [237] [DEBUG] [app] Ending request: urn:request:829da946-39a6-4ab2-bd6e-b8bd37c627f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:829da946-39a6-4ab2-bd6e-b8bd37c627f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:17:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:17:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:17:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:29,255 [235] [DEBUG] [app] Ending request: urn:request:7fcd115f-cbf3-4aab-9c99-055553e0828f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7fcd115f-cbf3-4aab-9c99-055553e0828f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:29,255 [238] [DEBUG] [app] Ending request: urn:request:e99193cc-0a53-4620-8e02-52b48069e337 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e99193cc-0a53-4620-8e02-52b48069e337', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:17:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
storagereplication stdout | 2025-02-07 05:17:31,239 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:17:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:17:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:32.625945+00:00 (in 1.004356 seconds)
namespacegcworker stdout | 2025-02-07 05:17:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:31 UTC)" (scheduled at 2025-02-07 05:17:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:17:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:17:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 31, 621862), True, datetime.datetime(2025, 2, 7, 5, 17, 31, 621862), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:17:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:17:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:17:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:17:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:17:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:31.621131+00:00 (in 58.994722 seconds)
namespacegcworker stdout | 2025-02-07 05:17:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:32 UTC)" (scheduled at 2025-02-07 05:17:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:17:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:17:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:17:33,024 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 05:17:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:38.732147+00:00 (in 4.917554 seconds)
notificationworker stdout | 2025-02-07 05:17:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:22:33 UTC)" (scheduled at 2025-02-07 05:17:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:17:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 17, 33, 814839), 'notification/%'])
notificationworker stdout | 2025-02-07 05:17:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 17, 33, 814839), True, datetime.datetime(2025, 2, 7, 5, 17, 33, 814839), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:17:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 17, 33, 814839), True, datetime.datetime(2025, 2, 7, 5, 17, 33, 814839), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 17, 33, 814839), 'notification/%'])
notificationworker stdout | 2025-02-07 05:17:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:17:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:22:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:17:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:17:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:03.919286+00:00 (in 29.999561 seconds)
buildlogsarchiver stdout | 2025-02-07 05:17:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:03 UTC)" (scheduled at 2025-02-07 05:17:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:17:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 17, 33, 919911), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:17:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:17:33,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:17:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:17:34,172 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:56.713547+00:00 (in 18.997178 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:37 UTC)" (scheduled at 2025-02-07 05:17:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:17:38,444 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:17:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:48.732147+00:00 (in 9.999579 seconds)
notificationworker stdout | 2025-02-07 05:17:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:48 UTC)" (scheduled at 2025-02-07 05:17:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:17:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:17:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 38, 732879), True, datetime.datetime(2025, 2, 7, 5, 17, 38, 732879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:17:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:17:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:17:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:17:39,471 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:17:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:17:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:54.423055+00:00 (in 13.998370 seconds)
gcworker stdout | 2025-02-07 05:17:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:10 UTC)" (scheduled at 2025-02-07 05:17:40.424250+00:00)
gcworker stdout | 2025-02-07 05:17:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:17:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905160431, None, 1, 0])
gcworker stdout | 2025-02-07 05:17:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:17:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:17:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:17:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:11.831521+00:00 (in 29.999573 seconds)
autopruneworker stdout | 2025-02-07 05:17:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:11 UTC)" (scheduled at 2025-02-07 05:17:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:17:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901861836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:17:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:17:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:17:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:17:43,391 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:17:43,937 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:17:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:17:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:59.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 05:17:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:59 UTC)" (scheduled at 2025-02-07 05:17:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:17:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:17:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:17:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:17:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:17:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:17:44,235 [237] [DEBUG] [app] Starting request: urn:request:513a2fa4-089e-4d68-9b10-0136e5782d1d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:44,235 [235] [DEBUG] [app] Starting request: urn:request:bb508f9a-5da9-48c2-92c7-15243460fd7e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:17:44,238 [239] [DEBUG] [app] Starting request: urn:request:59fd03bc-d06c-46d3-b23d-079ea59d8dc2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:44,238 [244] [DEBUG] [app] Starting request: urn:request:d4b07eb9-cf70-44c2-9211-a69ae012b51b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:44,239 [239] [DEBUG] [app] Ending request: urn:request:59fd03bc-d06c-46d3-b23d-079ea59d8dc2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59fd03bc-d06c-46d3-b23d-079ea59d8dc2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:17:44,239 [244] [DEBUG] [app] Ending request: urn:request:d4b07eb9-cf70-44c2-9211-a69ae012b51b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d4b07eb9-cf70-44c2-9211-a69ae012b51b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:17:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:17:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:44,242 [237] [DEBUG] [app] Starting request: urn:request:2f744ce9-e218-43da-8524-0ea3dc091767 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:44,242 [235] [DEBUG] [app] Starting request: urn:request:89c3d449-9b18-43e3-a1d7-650f74a1660c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:44,242 [237] [DEBUG] [app] Ending request: urn:request:2f744ce9-e218-43da-8524-0ea3dc091767 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f744ce9-e218-43da-8524-0ea3dc091767', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:17:44,242 [235] [DEBUG] [app] Ending request: urn:request:89c3d449-9b18-43e3-a1d7-650f74a1660c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89c3d449-9b18-43e3-a1d7-650f74a1660c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:17:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:44,254 [235] [DEBUG] [app] Ending request: urn:request:bb508f9a-5da9-48c2-92c7-15243460fd7e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb508f9a-5da9-48c2-92c7-15243460fd7e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:44,254 [237] [DEBUG] [app] Ending request: urn:request:513a2fa4-089e-4d68-9b10-0136e5782d1d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:513a2fa4-089e-4d68-9b10-0136e5782d1d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:17:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
logrotateworker stdout | 2025-02-07 05:17:44,567 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:17:45,960 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:17:46,744 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:17:46,819 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:17:47,565 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:17:47,567 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:17:47,570 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:17:47,573 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:17:47,575 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:17:48,078 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:48,401 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:17:48,586 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:17:48,589 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:17:48,592 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:17:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:50.814677+00:00 (in 2.082089 seconds)
notificationworker stdout | 2025-02-07 05:17:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:58 UTC)" (scheduled at 2025-02-07 05:17:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:17:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:17:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 48, 732862), True, datetime.datetime(2025, 2, 7, 5, 17, 48, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:17:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:17:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:17:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:17:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:17:48,826 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:17:48,907 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:17:49,160 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:17:49,535 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:17:49,882 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:17:49,949 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:17:49,954 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:17:49,958 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:17:49,965 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:17:49,967 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:17:49,971 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:17:49,977 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:17:49,979 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:17:49,982 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
securityscanningnotificationworker stdout | 2025-02-07 05:17:50,101 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:17:50,331 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:17:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:17:58.732147+00:00 (in 7.917020 seconds)
notificationworker stdout | 2025-02-07 05:17:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:50 UTC)" (scheduled at 2025-02-07 05:17:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:17:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:17:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:17:51,727 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:17:51,962 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:17:52,301 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:17:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:17:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:10.424250+00:00 (in 16.000755 seconds)
gcworker stdout | 2025-02-07 05:17:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:24 UTC)" (scheduled at 2025-02-07 05:17:54.423055+00:00)
gcworker stdout | 2025-02-07 05:17:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:17:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:17:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:17:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:56.118855+00:00 (in 59.999508 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:17:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:56 UTC)" (scheduled at 2025-02-07 05:17:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:17:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:17:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:17:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:37.715878+00:00 (in 41.001876 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:56 UTC)" (scheduled at 2025-02-07 05:17:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:17:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:17:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:17:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:08.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 05:17:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:08 UTC)" (scheduled at 2025-02-07 05:17:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:17:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:17:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 17, 58, 732853), True, datetime.datetime(2025, 2, 7, 5, 17, 58, 732853), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:17:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:17:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:17:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:17:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:17:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:14.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 05:17:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:14 UTC)" (scheduled at 2025-02-07 05:17:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:17:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:17:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:17:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:17:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:17:59,236 [235] [DEBUG] [app] Starting request: urn:request:649d3682-680b-49da-968c-31a773607517 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:59,236 [238] [DEBUG] [app] Starting request: urn:request:9db531cc-89da-4308-82af-5440f95d6cf0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:17:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:17:59,238 [243] [DEBUG] [app] Starting request: urn:request:1d35ca5a-db2e-4dea-bfc3-7d165977aa9a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:59,238 [239] [DEBUG] [app] Starting request: urn:request:5603043c-e466-4f7b-991b-b056cdcccba4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:17:59,239 [243] [DEBUG] [app] Ending request: urn:request:1d35ca5a-db2e-4dea-bfc3-7d165977aa9a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1d35ca5a-db2e-4dea-bfc3-7d165977aa9a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:17:59,239 [239] [DEBUG] [app] Ending request: urn:request:5603043c-e466-4f7b-991b-b056cdcccba4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5603043c-e466-4f7b-991b-b056cdcccba4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:17:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:17:59,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:17:59,241 [237] [DEBUG] [app] Starting request: urn:request:e71c366d-cc6f-44a2-9eb2-f3b7bf2bc222 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:59,242 [237] [DEBUG] [app] Ending request: urn:request:e71c366d-cc6f-44a2-9eb2-f3b7bf2bc222 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e71c366d-cc6f-44a2-9eb2-f3b7bf2bc222', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:17:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:17:59,242 [235] [DEBUG] [app] Starting request: urn:request:c83663b4-2ffd-4b7f-a15a-a1fd200d1bf4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:17:59,243 [235] [DEBUG] [app] Ending request: urn:request:c83663b4-2ffd-4b7f-a15a-a1fd200d1bf4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c83663b4-2ffd-4b7f-a15a-a1fd200d1bf4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:17:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:17:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:17:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:17:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:17:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:17:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:17:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:17:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:17:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:17:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:17:59,254 [235] [DEBUG] [app] Ending request: urn:request:649d3682-680b-49da-968c-31a773607517 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:649d3682-680b-49da-968c-31a773607517', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:17:59,255 [238] [DEBUG] [app] Ending request: urn:request:9db531cc-89da-4308-82af-5440f95d6cf0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9db531cc-89da-4308-82af-5440f95d6cf0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:17:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:17:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:17:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:17:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:17:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:17:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:04.936688+00:00 (in 4.998281 seconds)
exportactionlogsworker stdout | 2025-02-07 05:17:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:59 UTC)" (scheduled at 2025-02-07 05:17:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:17:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:17:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:18:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:18:01,275 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:18:03,058 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:18:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:18:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:23.420886+00:00 (in 20.001332 seconds)
repositorygcworker stdout | 2025-02-07 05:18:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:03 UTC)" (scheduled at 2025-02-07 05:18:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:18:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:18:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 3, 419864), True, datetime.datetime(2025, 2, 7, 5, 18, 3, 419864), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:18:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:18:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:18:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:18:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:18:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:33.919286+00:00 (in 29.999568 seconds)
buildlogsarchiver stdout | 2025-02-07 05:18:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:33 UTC)" (scheduled at 2025-02-07 05:18:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:18:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 18, 3, 919896), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:18:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:18:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:18:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:18:04,208 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:18:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:18:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:59.937899+00:00 (in 55.000746 seconds)
exportactionlogsworker stdout | 2025-02-07 05:18:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:04 UTC)" (scheduled at 2025-02-07 05:18:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:18:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:18:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 4, 937394), True, datetime.datetime(2025, 2, 7, 5, 18, 4, 937394), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:18:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:18:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:18:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:18:08,468 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:18:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:18:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:18.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 05:18:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:18 UTC)" (scheduled at 2025-02-07 05:18:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:18:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:18:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 8, 732854), True, datetime.datetime(2025, 2, 7, 5, 18, 8, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:18:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:18:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:18:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:18:09,479 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:18:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:18:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:24.423055+00:00 (in 13.998358 seconds)
gcworker stdout | 2025-02-07 05:18:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:40 UTC)" (scheduled at 2025-02-07 05:18:10.424250+00:00)
gcworker stdout | 2025-02-07 05:18:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:18:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905190431, None, 1, 0])
gcworker stdout | 2025-02-07 05:18:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:18:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:18:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:18:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:41.831521+00:00 (in 29.999557 seconds)
autopruneworker stdout | 2025-02-07 05:18:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:41 UTC)" (scheduled at 2025-02-07 05:18:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:18:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901891836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:18:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:18:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:18:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:18:13,426 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:18:13,970 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:18:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:18:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:29.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 05:18:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:29 UTC)" (scheduled at 2025-02-07 05:18:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:18:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:18:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:18:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:18:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:18:14,235 [237] [DEBUG] [app] Starting request: urn:request:5157d88b-56e0-42e8-b62e-fd11569b1dca (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:14,235 [235] [DEBUG] [app] Starting request: urn:request:e55a6735-3b26-4121-ba72-5f0ce0b42381 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:18:14,238 [245] [DEBUG] [app] Starting request: urn:request:b35015ea-ee7b-4fe3-af82-27bbb714576b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:14,238 [246] [DEBUG] [app] Starting request: urn:request:9a94ca61-1271-44bf-9445-c1d0cc472805 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:14,238 [245] [DEBUG] [app] Ending request: urn:request:b35015ea-ee7b-4fe3-af82-27bbb714576b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b35015ea-ee7b-4fe3-af82-27bbb714576b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:18:14,239 [246] [DEBUG] [app] Ending request: urn:request:9a94ca61-1271-44bf-9445-c1d0cc472805 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a94ca61-1271-44bf-9445-c1d0cc472805', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:18:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:18:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:18:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:14,241 [237] [DEBUG] [app] Starting request: urn:request:03418b2d-56de-46ff-8f2d-703c99259288 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:14,241 [237] [DEBUG] [app] Ending request: urn:request:03418b2d-56de-46ff-8f2d-703c99259288 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03418b2d-56de-46ff-8f2d-703c99259288', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:18:14,241 [238] [DEBUG] [app] Starting request: urn:request:2f2a14d2-4d10-438e-8f8d-3bec3b7cec5b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:18:14,242 [238] [DEBUG] [app] Ending request: urn:request:2f2a14d2-4d10-438e-8f8d-3bec3b7cec5b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f2a14d2-4d10-438e-8f8d-3bec3b7cec5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:18:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:18:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:18:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:14,253 [237] [DEBUG] [app] Ending request: urn:request:5157d88b-56e0-42e8-b62e-fd11569b1dca (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5157d88b-56e0-42e8-b62e-fd11569b1dca', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:14,253 [235] [DEBUG] [app] Ending request: urn:request:e55a6735-3b26-4121-ba72-5f0ce0b42381 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e55a6735-3b26-4121-ba72-5f0ce0b42381', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:18:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:18:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:18:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
logrotateworker stdout | 2025-02-07 05:18:14,593 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:18:15,996 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:18:16,780 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:18:16,831 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:18:17,572 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:18:17,575 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:18:17,579 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:18:17,582 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:18:17,584 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:18:18,106 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:18,426 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:18:18,595 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:18:18,599 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:18:18,602 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:18:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:18:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:28.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:18:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:28 UTC)" (scheduled at 2025-02-07 05:18:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:18:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:18:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 18, 732877), True, datetime.datetime(2025, 2, 7, 5, 18, 18, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:18:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:18:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:18:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:18:18,862 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:18:18,914 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:18:19,196 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:18:19,572 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:18:19,917 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:18:19,961 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:18:19,964 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:18:19,967 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:18:19,973 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:18:19,976 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:18:19,979 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:18:19,985 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:18:19,988 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:18:19,990 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
securityscanningnotificationworker stdout | 2025-02-07 05:18:20,138 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:18:20,367 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:18:21,754 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:18:21,968 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:18:22,337 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:18:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:18:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:32.420707+00:00 (in 8.999356 seconds)
repositorygcworker stdout | 2025-02-07 05:18:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:23 UTC)" (scheduled at 2025-02-07 05:18:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:18:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:18:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:18:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:18:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:40.424250+00:00 (in 16.000739 seconds)
gcworker stdout | 2025-02-07 05:18:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:54 UTC)" (scheduled at 2025-02-07 05:18:24.423055+00:00)
gcworker stdout | 2025-02-07 05:18:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:18:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:18:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:18:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:18:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:38.732147+00:00 (in 9.999511 seconds)
notificationworker stdout | 2025-02-07 05:18:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:38 UTC)" (scheduled at 2025-02-07 05:18:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:18:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:18:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 28, 732948), True, datetime.datetime(2025, 2, 7, 5, 18, 28, 732948), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:18:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:18:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:18:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:18:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:18:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:44.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 05:18:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:44 UTC)" (scheduled at 2025-02-07 05:18:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:18:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:18:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:18:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:18:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:18:29,236 [237] [DEBUG] [app] Starting request: urn:request:8d2e9669-d79f-43a8-91d6-73e8890ba3c1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:29,236 [238] [DEBUG] [app] Starting request: urn:request:e25aef12-de9c-4e25-8901-d99c8425a4d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:29,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:18:29,240 [246] [DEBUG] [app] Starting request: urn:request:231d6f2a-1739-49f3-8a9b-7707c9903c66 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:29,240 [239] [DEBUG] [app] Starting request: urn:request:8d043a0b-ff4d-4d1a-8258-0d6d15440a65 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:29,240 [246] [DEBUG] [app] Ending request: urn:request:231d6f2a-1739-49f3-8a9b-7707c9903c66 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:231d6f2a-1739-49f3-8a9b-7707c9903c66', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:18:29,240 [239] [DEBUG] [app] Ending request: urn:request:8d043a0b-ff4d-4d1a-8258-0d6d15440a65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d043a0b-ff4d-4d1a-8258-0d6d15440a65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:18:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:18:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:29,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:29,243 [236] [DEBUG] [app] Starting request: urn:request:2cfc3899-a5d0-4604-bab8-05a8a3064fbc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:29,243 [235] [DEBUG] [app] Starting request: urn:request:fad7ec6e-92b3-4af8-a90b-623904461b68 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:29,243 [236] [DEBUG] [app] Ending request: urn:request:2cfc3899-a5d0-4604-bab8-05a8a3064fbc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2cfc3899-a5d0-4604-bab8-05a8a3064fbc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:18:29,243 [235] [DEBUG] [app] Ending request: urn:request:fad7ec6e-92b3-4af8-a90b-623904461b68 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fad7ec6e-92b3-4af8-a90b-623904461b68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:18:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:18:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:18:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:18:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:18:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:29,255 [238] [DEBUG] [app] Ending request: urn:request:e25aef12-de9c-4e25-8901-d99c8425a4d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e25aef12-de9c-4e25-8901-d99c8425a4d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:18:29,255 [237] [DEBUG] [app] Ending request: urn:request:8d2e9669-d79f-43a8-91d6-73e8890ba3c1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8d2e9669-d79f-43a8-91d6-73e8890ba3c1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:18:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:18:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 05:18:31,311 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:18:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:18:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:32.625945+00:00 (in 1.004353 seconds)
namespacegcworker stdout | 2025-02-07 05:18:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:31 UTC)" (scheduled at 2025-02-07 05:18:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:18:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:18:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 31, 621797), True, datetime.datetime(2025, 2, 7, 5, 18, 31, 621797), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:18:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:18:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:18:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:18:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:18:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:03.419110+00:00 (in 30.997955 seconds)
repositorygcworker stdout | 2025-02-07 05:18:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:23:32 UTC)" (scheduled at 2025-02-07 05:18:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:18:32,422 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 18, 32, 421467), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:18:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 18, 32, 421467), True, datetime.datetime(2025, 2, 7, 5, 18, 32, 421467), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:18:32,431 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 18, 32, 421467), True, datetime.datetime(2025, 2, 7, 5, 18, 32, 421467), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 18, 32, 421467), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:18:32,432 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:18:32,432 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:23:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:18:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:18:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:31.621131+00:00 (in 58.994735 seconds)
namespacegcworker stdout | 2025-02-07 05:18:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:32 UTC)" (scheduled at 2025-02-07 05:18:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:18:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:18:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:18:33,082 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:18:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:18:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:03.919286+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2025-02-07 05:18:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:03 UTC)" (scheduled at 2025-02-07 05:18:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:18:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 18, 33, 920093), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:18:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:18:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:18:33,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:18:34,230 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:56.713547+00:00 (in 18.997155 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:37 UTC)" (scheduled at 2025-02-07 05:18:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:18:38,504 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:18:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:18:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:48.732147+00:00 (in 9.999514 seconds)
notificationworker stdout | 2025-02-07 05:18:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:48 UTC)" (scheduled at 2025-02-07 05:18:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:18:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:18:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 38, 732913), True, datetime.datetime(2025, 2, 7, 5, 18, 38, 732913), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:18:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:18:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:18:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:18:39,507 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:18:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:18:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:54.423055+00:00 (in 13.998362 seconds)
gcworker stdout | 2025-02-07 05:18:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:10 UTC)" (scheduled at 2025-02-07 05:18:40.424250+00:00)
gcworker stdout | 2025-02-07 05:18:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:18:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905220431, None, 1, 0])
gcworker stdout | 2025-02-07 05:18:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:18:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:18:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:18:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:11.831521+00:00 (in 29.999535 seconds)
autopruneworker stdout | 2025-02-07 05:18:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:11 UTC)" (scheduled at 2025-02-07 05:18:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:18:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901921836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:18:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:18:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:18:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:18:43,463 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:18:44,006 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:18:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:18:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:59.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 05:18:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:59 UTC)" (scheduled at 2025-02-07 05:18:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:18:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:18:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:18:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:18:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:18:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:18:44,236 [235] [DEBUG] [app] Starting request: urn:request:5bd29c2d-2f03-4225-ab3f-b11c8846afa4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:44,236 [238] [DEBUG] [app] Starting request: urn:request:80d94c63-b682-4536-bd4e-d9935cae134c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:18:44,238 [246] [DEBUG] [app] Starting request: urn:request:a2ccb03d-17a1-4506-b3cb-35af6d937b09 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:44,238 [239] [DEBUG] [app] Starting request: urn:request:8030e610-944d-45a5-8ae8-d8101af1c52c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:44,239 [239] [DEBUG] [app] Ending request: urn:request:8030e610-944d-45a5-8ae8-d8101af1c52c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8030e610-944d-45a5-8ae8-d8101af1c52c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:18:44,239 [246] [DEBUG] [app] Ending request: urn:request:a2ccb03d-17a1-4506-b3cb-35af6d937b09 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2ccb03d-17a1-4506-b3cb-35af6d937b09', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:18:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:18:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:44,242 [236] [DEBUG] [app] Starting request: urn:request:f6be6753-6b60-423c-a08e-34cfd25f9f54 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:44,242 [237] [DEBUG] [app] Starting request: urn:request:006ef338-d1c3-47b3-abb3-abffd0e1e65e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:44,242 [236] [DEBUG] [app] Ending request: urn:request:f6be6753-6b60-423c-a08e-34cfd25f9f54 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6be6753-6b60-423c-a08e-34cfd25f9f54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:18:44,242 [237] [DEBUG] [app] Ending request: urn:request:006ef338-d1c3-47b3-abb3-abffd0e1e65e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:006ef338-d1c3-47b3-abb3-abffd0e1e65e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:18:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:18:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:18:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:44,254 [238] [DEBUG] [app] Ending request: urn:request:80d94c63-b682-4536-bd4e-d9935cae134c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:80d94c63-b682-4536-bd4e-d9935cae134c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:44,254 [235] [DEBUG] [app] Ending request: urn:request:5bd29c2d-2f03-4225-ab3f-b11c8846afa4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5bd29c2d-2f03-4225-ab3f-b11c8846afa4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:18:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:18:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:18:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
logrotateworker stdout | 2025-02-07 05:18:44,629 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:18:46,032 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:18:46,799 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:18:46,867 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:18:47,580 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:18:47,583 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:18:47,586 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:18:47,590 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:18:47,593 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:18:48,138 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:48,442 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:18:48,603 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:18:48,606 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:18:48,609 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:18:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:18:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:50.814677+00:00 (in 2.082091 seconds)
notificationworker stdout | 2025-02-07 05:18:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:58 UTC)" (scheduled at 2025-02-07 05:18:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:18:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:18:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 48, 732856), True, datetime.datetime(2025, 2, 7, 5, 18, 48, 732856), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:18:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:18:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:18:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:18:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:18:48,872 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:18:48,951 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:18:49,215 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:18:49,579 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:18:49,932 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:18:49,973 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:18:49,976 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:18:49,979 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:18:49,982 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:18:49,985 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:18:49,988 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:18:49,995 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:18:49,999 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:18:50,002 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:18:50,174 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:18:50,401 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:18:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:18:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:18:58.732147+00:00 (in 7.917009 seconds)
notificationworker stdout | 2025-02-07 05:18:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:50 UTC)" (scheduled at 2025-02-07 05:18:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:18:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:18:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:18:51,790 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:18:52,005 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:18:52,373 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:18:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:18:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:10.424250+00:00 (in 16.000729 seconds)
gcworker stdout | 2025-02-07 05:18:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:24 UTC)" (scheduled at 2025-02-07 05:18:54.423055+00:00)
gcworker stdout | 2025-02-07 05:18:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:18:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:18:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:18:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:56.118855+00:00 (in 59.999521 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:18:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:56 UTC)" (scheduled at 2025-02-07 05:18:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:18:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:18:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:18:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:37.715878+00:00 (in 41.001851 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:56 UTC)" (scheduled at 2025-02-07 05:18:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:18:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:18:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:18:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:08.732147+00:00 (in 9.999573 seconds)
notificationworker stdout | 2025-02-07 05:18:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:08 UTC)" (scheduled at 2025-02-07 05:18:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:18:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:18:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 18, 58, 732841), True, datetime.datetime(2025, 2, 7, 5, 18, 58, 732841), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:18:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:18:58,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:18:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:18:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:18:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:14.225378+00:00 (in 14.999573 seconds)
quotatotalworker stdout | 2025-02-07 05:18:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:14 UTC)" (scheduled at 2025-02-07 05:18:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:18:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:18:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:18:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:18:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:18:59,235 [236] [DEBUG] [app] Starting request: urn:request:8585adfd-6a49-4cae-89ef-2c42c07e1f25 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:59,235 [238] [DEBUG] [app] Starting request: urn:request:69f6ef23-c99b-440d-ad53-1231165b05f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:18:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:18:59,238 [239] [DEBUG] [app] Starting request: urn:request:0ab7e5b2-4328-49f5-914e-a9ab461aa5d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:59,238 [246] [DEBUG] [app] Starting request: urn:request:4ddd54bd-5375-42f1-8fa0-a535f7a8a4a1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:18:59,239 [239] [DEBUG] [app] Ending request: urn:request:0ab7e5b2-4328-49f5-914e-a9ab461aa5d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0ab7e5b2-4328-49f5-914e-a9ab461aa5d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:18:59,239 [246] [DEBUG] [app] Ending request: urn:request:4ddd54bd-5375-42f1-8fa0-a535f7a8a4a1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4ddd54bd-5375-42f1-8fa0-a535f7a8a4a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:18:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:18:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:18:59,242 [235] [DEBUG] [app] Starting request: urn:request:2dae4b0f-4de8-4939-93ae-c5e171dc10a6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:59,242 [237] [DEBUG] [app] Starting request: urn:request:b4961806-2138-43f9-8792-ea96f3c5c39e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:18:59,243 [237] [DEBUG] [app] Ending request: urn:request:b4961806-2138-43f9-8792-ea96f3c5c39e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4961806-2138-43f9-8792-ea96f3c5c39e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:18:59,243 [235] [DEBUG] [app] Ending request: urn:request:2dae4b0f-4de8-4939-93ae-c5e171dc10a6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2dae4b0f-4de8-4939-93ae-c5e171dc10a6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:18:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:18:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:18:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:18:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:18:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:18:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:18:59,245 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2025, 2, 7, 5, 18, 59, 244441), None, 'quay', 'quay', datetime.datetime(2025, 2, 7, 5, 18, 59, 244468), 'quay', None, datetime.datetime(2025, 2, 6, 5, 18, 59, 244485), datetime.datetime(2025, 1, 31, 5, 18, 59, 244502), None])
gunicorn-web stdout | 2025-02-07 05:18:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:59,251 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:18:59,251 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:18:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:59,254 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:18:59,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:18:59,255 [238] [DEBUG] [app] Ending request: urn:request:69f6ef23-c99b-440d-ad53-1231165b05f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:69f6ef23-c99b-440d-ad53-1231165b05f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:18:59,255 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:18:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
gunicorn-web stdout | 2025-02-07 05:18:59,256 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:18:59,257 [236] [DEBUG] [app] Ending request: urn:request:8585adfd-6a49-4cae-89ef-2c42c07e1f25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8585adfd-6a49-4cae-89ef-2c42c07e1f25', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:18:59,258 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:18:59,258 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:18:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:18:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.023)
exportactionlogsworker stdout | 2025-02-07 05:18:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:18:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:04.936688+00:00 (in 4.998298 seconds)
exportactionlogsworker stdout | 2025-02-07 05:18:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:59 UTC)" (scheduled at 2025-02-07 05:18:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:18:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:18:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:19:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:19:01,347 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:19:03,117 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:19:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:19:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:23.420886+00:00 (in 20.001309 seconds)
repositorygcworker stdout | 2025-02-07 05:19:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:03 UTC)" (scheduled at 2025-02-07 05:19:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:19:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:19:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 3, 419846), True, datetime.datetime(2025, 2, 7, 5, 19, 3, 419846), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:19:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:19:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:19:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:19:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:19:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:33.919286+00:00 (in 29.999518 seconds)
buildlogsarchiver stdout | 2025-02-07 05:19:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:33 UTC)" (scheduled at 2025-02-07 05:19:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:19:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 19, 3, 920013), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:19:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:19:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:19:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:19:04,266 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:19:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:19:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:52.937648+00:00 (in 48.000507 seconds)
exportactionlogsworker stdout | 2025-02-07 05:19:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:04 UTC)" (scheduled at 2025-02-07 05:19:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:19:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:19:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 4, 937487), True, datetime.datetime(2025, 2, 7, 5, 19, 4, 937487), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:19:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:19:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:19:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:19:08,540 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:19:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:19:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:18.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 05:19:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:18 UTC)" (scheduled at 2025-02-07 05:19:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:19:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:19:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 8, 732906), True, datetime.datetime(2025, 2, 7, 5, 19, 8, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:19:08,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:19:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:19:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:19:09,543 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:19:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:19:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:24.423055+00:00 (in 13.998347 seconds)
gcworker stdout | 2025-02-07 05:19:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:40 UTC)" (scheduled at 2025-02-07 05:19:10.424250+00:00)
gcworker stdout | 2025-02-07 05:19:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:19:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905250431, None, 1, 0])
gcworker stdout | 2025-02-07 05:19:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:19:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:19:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:19:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:41.831521+00:00 (in 29.999517 seconds)
autopruneworker stdout | 2025-02-07 05:19:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:41 UTC)" (scheduled at 2025-02-07 05:19:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:19:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901951836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:19:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:19:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:19:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:19:13,499 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:19:14,017 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:19:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:19:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:29.225378+00:00 (in 14.999583 seconds)
quotatotalworker stdout | 2025-02-07 05:19:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:29 UTC)" (scheduled at 2025-02-07 05:19:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:19:14,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:19:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:19:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:19:14,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:19:14,236 [237] [DEBUG] [app] Starting request: urn:request:10feb273-055c-493c-9c72-cd50c6cbff3e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:14,236 [235] [DEBUG] [app] Starting request: urn:request:1887a2cb-fa85-4361-b8cc-d9e8922af602 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:19:14,239 [245] [DEBUG] [app] Starting request: urn:request:e80c0de7-6922-4921-b88f-0715048eb404 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:14,239 [239] [DEBUG] [app] Starting request: urn:request:8fcd25ae-7adb-4b98-b8d8-f59653cef3d0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:14,239 [239] [DEBUG] [app] Ending request: urn:request:8fcd25ae-7adb-4b98-b8d8-f59653cef3d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fcd25ae-7adb-4b98-b8d8-f59653cef3d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:19:14,239 [245] [DEBUG] [app] Ending request: urn:request:e80c0de7-6922-4921-b88f-0715048eb404 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e80c0de7-6922-4921-b88f-0715048eb404', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:19:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:19:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:19:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:14,242 [235] [DEBUG] [app] Starting request: urn:request:deb131dd-85be-493a-b5ba-6f683db3e565 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:14,242 [238] [DEBUG] [app] Starting request: urn:request:93df842e-a924-4cc6-a50d-118572b24436 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:14,242 [235] [DEBUG] [app] Ending request: urn:request:deb131dd-85be-493a-b5ba-6f683db3e565 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:deb131dd-85be-493a-b5ba-6f683db3e565', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:19:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:14,242 [238] [DEBUG] [app] Ending request: urn:request:93df842e-a924-4cc6-a50d-118572b24436 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93df842e-a924-4cc6-a50d-118572b24436', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:19:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:19:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:14,254 [235] [DEBUG] [app] Ending request: urn:request:1887a2cb-fa85-4361-b8cc-d9e8922af602 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1887a2cb-fa85-4361-b8cc-d9e8922af602', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:19:14,254 [237] [DEBUG] [app] Ending request: urn:request:10feb273-055c-493c-9c72-cd50c6cbff3e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10feb273-055c-493c-9c72-cd50c6cbff3e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:19:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:19:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
logrotateworker stdout | 2025-02-07 05:19:14,650 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:19:16,069 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:19:16,831 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:19:16,901 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:19:17,588 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:19:17,590 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:19:17,594 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:19:17,597 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:19:17,599 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:19:18,174 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:18,478 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:19:18,611 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:19:18,615 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:19:18,618 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:19:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:19:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:28.732147+00:00 (in 9.999501 seconds)
notificationworker stdout | 2025-02-07 05:19:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:28 UTC)" (scheduled at 2025-02-07 05:19:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:19:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:19:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 18, 732963), True, datetime.datetime(2025, 2, 7, 5, 19, 18, 732963), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:19:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:19:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:19:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:19:18,906 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:19:18,977 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:19:19,249 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:19:19,588 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:19:19,958 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:19:19,984 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:19:19,988 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:19:19,992 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:19:19,995 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:19:19,998 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:19:20,001 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:19:20,004 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:19:20,007 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:19:20,013 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:19:20,207 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:19:20,437 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:19:21,827 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:19:22,041 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:19:22,410 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:19:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:19:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:03.419110+00:00 (in 39.997743 seconds)
repositorygcworker stdout | 2025-02-07 05:19:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:23 UTC)" (scheduled at 2025-02-07 05:19:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:19:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:19:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:19:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:19:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:40.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 05:19:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:54 UTC)" (scheduled at 2025-02-07 05:19:24.423055+00:00)
gcworker stdout | 2025-02-07 05:19:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:19:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:19:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:19:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:19:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:38.732147+00:00 (in 9.999507 seconds)
notificationworker stdout | 2025-02-07 05:19:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:38 UTC)" (scheduled at 2025-02-07 05:19:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:19:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:19:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 28, 732924), True, datetime.datetime(2025, 2, 7, 5, 19, 28, 732924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:19:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:19:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:19:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:19:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:19:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:44.225378+00:00 (in 14.999568 seconds)
quotatotalworker stdout | 2025-02-07 05:19:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:44 UTC)" (scheduled at 2025-02-07 05:19:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:19:29,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:19:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:19:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:19:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:19:29,235 [237] [DEBUG] [app] Starting request: urn:request:6faeaeee-cebb-4ea3-89f5-f59e22f157e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:29,235 [238] [DEBUG] [app] Starting request: urn:request:50f143d4-07f0-4e04-9bb7-c894fe05cdc6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:19:29,238 [246] [DEBUG] [app] Starting request: urn:request:2f5d61ff-79a7-4cda-a944-3e0f9a11b397 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:29,238 [239] [DEBUG] [app] Starting request: urn:request:35e41fc2-c777-49c0-b8e1-b4aae9034981 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:29,239 [239] [DEBUG] [app] Ending request: urn:request:35e41fc2-c777-49c0-b8e1-b4aae9034981 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35e41fc2-c777-49c0-b8e1-b4aae9034981', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:19:29,239 [246] [DEBUG] [app] Ending request: urn:request:2f5d61ff-79a7-4cda-a944-3e0f9a11b397 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f5d61ff-79a7-4cda-a944-3e0f9a11b397', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:19:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:19:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:29,242 [235] [DEBUG] [app] Starting request: urn:request:314cedd6-915e-45cc-8060-13215e64c1de (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:29,242 [238] [DEBUG] [app] Starting request: urn:request:cf7a0293-7bbc-4215-b4b2-6e4e1583a54f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:29,242 [238] [DEBUG] [app] Ending request: urn:request:cf7a0293-7bbc-4215-b4b2-6e4e1583a54f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cf7a0293-7bbc-4215-b4b2-6e4e1583a54f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:19:29,242 [235] [DEBUG] [app] Ending request: urn:request:314cedd6-915e-45cc-8060-13215e64c1de (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:314cedd6-915e-45cc-8060-13215e64c1de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:19:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:19:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:29,253 [237] [DEBUG] [app] Ending request: urn:request:6faeaeee-cebb-4ea3-89f5-f59e22f157e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6faeaeee-cebb-4ea3-89f5-f59e22f157e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:29,253 [238] [DEBUG] [app] Ending request: urn:request:50f143d4-07f0-4e04-9bb7-c894fe05cdc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:50f143d4-07f0-4e04-9bb7-c894fe05cdc6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:19:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:19:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:19:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
storagereplication stdout | 2025-02-07 05:19:31,379 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:19:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:19:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:32.625945+00:00 (in 1.004359 seconds)
namespacegcworker stdout | 2025-02-07 05:19:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:31 UTC)" (scheduled at 2025-02-07 05:19:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:19:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:19:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 31, 621864), True, datetime.datetime(2025, 2, 7, 5, 19, 31, 621864), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:19:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:19:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:19:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:19:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:19:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:31.621131+00:00 (in 58.994733 seconds)
namespacegcworker stdout | 2025-02-07 05:19:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:32 UTC)" (scheduled at 2025-02-07 05:19:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:19:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:19:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:19:33,154 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:19:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:19:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:03.919286+00:00 (in 29.999542 seconds)
buildlogsarchiver stdout | 2025-02-07 05:19:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:03 UTC)" (scheduled at 2025-02-07 05:19:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:19:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 19, 33, 920005), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:19:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:19:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:19:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:19:34,294 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:56.713547+00:00 (in 18.997186 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:37 UTC)" (scheduled at 2025-02-07 05:19:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:19:38,577 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:19:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:19:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:48.732147+00:00 (in 9.999524 seconds)
notificationworker stdout | 2025-02-07 05:19:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:48 UTC)" (scheduled at 2025-02-07 05:19:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:19:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:19:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 38, 732898), True, datetime.datetime(2025, 2, 7, 5, 19, 38, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:19:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:19:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:19:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:19:39,579 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:19:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:19:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:54.423055+00:00 (in 13.998348 seconds)
gcworker stdout | 2025-02-07 05:19:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:10 UTC)" (scheduled at 2025-02-07 05:19:40.424250+00:00)
gcworker stdout | 2025-02-07 05:19:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:19:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905280431, None, 1, 0])
gcworker stdout | 2025-02-07 05:19:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:19:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:19:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:19:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:11.831521+00:00 (in 29.999556 seconds)
autopruneworker stdout | 2025-02-07 05:19:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:11 UTC)" (scheduled at 2025-02-07 05:19:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:19:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738901981836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:19:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:19:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:19:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:19:43,505 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:19:44,054 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:19:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:19:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:59.225378+00:00 (in 14.999556 seconds)
quotatotalworker stdout | 2025-02-07 05:19:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:59 UTC)" (scheduled at 2025-02-07 05:19:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:19:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:19:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:19:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:19:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:19:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:19:44,235 [235] [DEBUG] [app] Starting request: urn:request:f2010a4a-d9be-4bee-ab4a-956d6dd85e81 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:44,235 [237] [DEBUG] [app] Starting request: urn:request:879d5289-f857-4f4e-b930-9da01e2a6389 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:19:44,238 [246] [DEBUG] [app] Starting request: urn:request:09570075-b5a4-48d1-b2ba-f714892e473a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:44,238 [239] [DEBUG] [app] Starting request: urn:request:04646e8e-ec22-4d9d-b052-b8339b4af8fb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:44,239 [246] [DEBUG] [app] Ending request: urn:request:09570075-b5a4-48d1-b2ba-f714892e473a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09570075-b5a4-48d1-b2ba-f714892e473a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:19:44,239 [239] [DEBUG] [app] Ending request: urn:request:04646e8e-ec22-4d9d-b052-b8339b4af8fb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:04646e8e-ec22-4d9d-b052-b8339b4af8fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:19:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:19:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:19:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:44,241 [237] [DEBUG] [app] Starting request: urn:request:1d5f3406-c2e1-46ed-898a-c51465862c03 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:44,242 [238] [DEBUG] [app] Starting request: urn:request:3ea7dac8-6d35-452e-8557-3a3d7ecdc3e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:44,242 [237] [DEBUG] [app] Ending request: urn:request:1d5f3406-c2e1-46ed-898a-c51465862c03 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d5f3406-c2e1-46ed-898a-c51465862c03', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:19:44,242 [238] [DEBUG] [app] Ending request: urn:request:3ea7dac8-6d35-452e-8557-3a3d7ecdc3e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ea7dac8-6d35-452e-8557-3a3d7ecdc3e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:19:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:19:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:19:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:44,253 [235] [DEBUG] [app] Ending request: urn:request:f2010a4a-d9be-4bee-ab4a-956d6dd85e81 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f2010a4a-d9be-4bee-ab4a-956d6dd85e81', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:19:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 05:19:44,253 [237] [DEBUG] [app] Ending request: urn:request:879d5289-f857-4f4e-b930-9da01e2a6389 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:879d5289-f857-4f4e-b930-9da01e2a6389', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:19:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
logrotateworker stdout | 2025-02-07 05:19:44,686 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:19:46,078 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:19:46,868 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:19:46,918 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:19:47,595 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:19:47,599 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:19:47,602 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:19:47,605 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:19:47,608 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:19:48,203 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:48,510 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:19:48,618 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:19:48,624 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:19:48,626 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:19:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:19:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:50.814677+00:00 (in 2.082102 seconds)
notificationworker stdout | 2025-02-07 05:19:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:58 UTC)" (scheduled at 2025-02-07 05:19:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:19:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:19:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 48, 732846), True, datetime.datetime(2025, 2, 7, 5, 19, 48, 732846), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:19:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:19:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:19:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:19:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:19:48,932 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:19:49,012 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:19:49,285 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:19:49,613 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:19:49,995 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:19:49,998 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:19:50,001 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:19:50,004 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:19:50,006 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:19:50,009 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:19:50,013 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:19:50,015 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:19:50,018 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:19:50,022 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:19:50,243 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:19:50,472 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:19:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:19:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:58.732147+00:00 (in 7.917016 seconds)
notificationworker stdout | 2025-02-07 05:19:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:50 UTC)" (scheduled at 2025-02-07 05:19:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:19:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:19:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:19:51,864 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:19:52,077 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:19:52,446 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
exportactionlogsworker stdout | 2025-02-07 05:19:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:19:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:19:59.937899+00:00 (in 6.999773 seconds)
exportactionlogsworker stdout | 2025-02-07 05:19:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:24:52 UTC)" (scheduled at 2025-02-07 05:19:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:19:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 19, 52, 938401), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:19:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 19, 52, 938401), True, datetime.datetime(2025, 2, 7, 5, 19, 52, 938401), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:19:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 19, 52, 938401), True, datetime.datetime(2025, 2, 7, 5, 19, 52, 938401), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 19, 52, 938401), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:19:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:19:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:24:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:19:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:19:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:10.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 05:19:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:24 UTC)" (scheduled at 2025-02-07 05:19:54.423055+00:00)
gcworker stdout | 2025-02-07 05:19:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:19:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:19:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:19:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:56.118855+00:00 (in 59.999529 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:19:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:56 UTC)" (scheduled at 2025-02-07 05:19:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:19:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:19:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:19:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:37.715878+00:00 (in 41.001867 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:56 UTC)" (scheduled at 2025-02-07 05:19:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:19:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:19:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:19:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:08.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:19:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:08 UTC)" (scheduled at 2025-02-07 05:19:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:19:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:19:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 19, 58, 732852), True, datetime.datetime(2025, 2, 7, 5, 19, 58, 732852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:19:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:19:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:19:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:19:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:19:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:14.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 05:19:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:14 UTC)" (scheduled at 2025-02-07 05:19:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:19:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:19:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:19:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:19:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:19:59,235 [238] [DEBUG] [app] Starting request: urn:request:4f59b2e5-5c48-47c5-a7b0-acf2336d91e2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:59,235 [235] [DEBUG] [app] Starting request: urn:request:61d53289-c277-4868-8b81-9df6f549e02a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:19:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:19:59,238 [239] [DEBUG] [app] Starting request: urn:request:a49077e9-86d1-40e8-a891-9b9ac41967a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:59,238 [246] [DEBUG] [app] Starting request: urn:request:36076a6b-6c56-4241-a216-558fe640abc6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:19:59,239 [246] [DEBUG] [app] Ending request: urn:request:36076a6b-6c56-4241-a216-558fe640abc6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36076a6b-6c56-4241-a216-558fe640abc6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:19:59,239 [239] [DEBUG] [app] Ending request: urn:request:a49077e9-86d1-40e8-a891-9b9ac41967a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a49077e9-86d1-40e8-a891-9b9ac41967a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:19:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:19:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:19:59,243 [237] [DEBUG] [app] Starting request: urn:request:cae1d1f0-728a-4f22-bfbe-f4c5f7e92385 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:59,243 [236] [DEBUG] [app] Starting request: urn:request:532d574a-173b-44a3-b13b-cf8e0dc80133 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:19:59,243 [237] [DEBUG] [app] Ending request: urn:request:cae1d1f0-728a-4f22-bfbe-f4c5f7e92385 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cae1d1f0-728a-4f22-bfbe-f4c5f7e92385', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:19:59,243 [236] [DEBUG] [app] Ending request: urn:request:532d574a-173b-44a3-b13b-cf8e0dc80133 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:532d574a-173b-44a3-b13b-cf8e0dc80133', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:19:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:19:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:19:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:19:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:19:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:19:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:19:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:19:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:19:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:19:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:19:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:19:59,255 [238] [DEBUG] [app] Ending request: urn:request:4f59b2e5-5c48-47c5-a7b0-acf2336d91e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f59b2e5-5c48-47c5-a7b0-acf2336d91e2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:59,255 [235] [DEBUG] [app] Ending request: urn:request:61d53289-c277-4868-8b81-9df6f549e02a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61d53289-c277-4868-8b81-9df6f549e02a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:19:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:19:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:19:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:19:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:19:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:19:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:19:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:19:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:04.936688+00:00 (in 4.998305 seconds)
exportactionlogsworker stdout | 2025-02-07 05:19:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:59 UTC)" (scheduled at 2025-02-07 05:19:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:19:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:19:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:20:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:20:01,407 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:20:03,190 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:20:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:20:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:23.420886+00:00 (in 20.001331 seconds)
repositorygcworker stdout | 2025-02-07 05:20:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:03 UTC)" (scheduled at 2025-02-07 05:20:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:20:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:20:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 3, 419865), True, datetime.datetime(2025, 2, 7, 5, 20, 3, 419865), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:20:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:20:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:20:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:20:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:20:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:33.919286+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2025-02-07 05:20:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:33 UTC)" (scheduled at 2025-02-07 05:20:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:20:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 20, 3, 919999), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:20:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:20:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:20:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:20:04,319 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:20:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:20:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:59.937899+00:00 (in 55.000747 seconds)
exportactionlogsworker stdout | 2025-02-07 05:20:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:04 UTC)" (scheduled at 2025-02-07 05:20:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:20:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:20:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 4, 937459), True, datetime.datetime(2025, 2, 7, 5, 20, 4, 937459), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:20:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:20:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:20:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:20:08,589 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:20:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:20:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:18.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 05:20:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:18 UTC)" (scheduled at 2025-02-07 05:20:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:20:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:20:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 8, 732860), True, datetime.datetime(2025, 2, 7, 5, 20, 8, 732860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:20:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:20:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:20:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:20:09,589 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:20:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:20:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:24.423055+00:00 (in 13.998365 seconds)
gcworker stdout | 2025-02-07 05:20:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:40 UTC)" (scheduled at 2025-02-07 05:20:10.424250+00:00)
gcworker stdout | 2025-02-07 05:20:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:20:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905310431, None, 1, 0])
gcworker stdout | 2025-02-07 05:20:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:20:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:20:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:20:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:41.831521+00:00 (in 29.999533 seconds)
autopruneworker stdout | 2025-02-07 05:20:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:41 UTC)" (scheduled at 2025-02-07 05:20:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:20:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902011836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:20:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:20:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:20:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:20:13,542 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:20:14,090 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:20:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:20:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:29.225378+00:00 (in 14.999528 seconds)
quotatotalworker stdout | 2025-02-07 05:20:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:29 UTC)" (scheduled at 2025-02-07 05:20:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:20:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:20:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:20:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:20:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:20:14,235 [237] [DEBUG] [app] Starting request: urn:request:42511798-f97f-4302-b70d-1cb4f356bdcc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:14,235 [235] [DEBUG] [app] Starting request: urn:request:f0d234e4-49f6-42eb-a421-a7cef7119abd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:20:14,238 [244] [DEBUG] [app] Starting request: urn:request:82ed89de-7143-4659-9c6d-3a02358a09e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:14,238 [246] [DEBUG] [app] Starting request: urn:request:a0d7eee2-fcea-454f-9414-6ec485b7cf60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:14,239 [246] [DEBUG] [app] Ending request: urn:request:a0d7eee2-fcea-454f-9414-6ec485b7cf60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0d7eee2-fcea-454f-9414-6ec485b7cf60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:20:14,239 [244] [DEBUG] [app] Ending request: urn:request:82ed89de-7143-4659-9c6d-3a02358a09e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82ed89de-7143-4659-9c6d-3a02358a09e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:20:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:20:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:14,243 [236] [DEBUG] [app] Starting request: urn:request:a9a4679d-484b-4cb2-a540-7978e8f16c3e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:20:14,243 [238] [DEBUG] [app] Starting request: urn:request:c0c5234f-d96f-4a15-9ce6-81ca0b2f0e45 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:20:14,243 [238] [DEBUG] [app] Ending request: urn:request:c0c5234f-d96f-4a15-9ce6-81ca0b2f0e45 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c0c5234f-d96f-4a15-9ce6-81ca0b2f0e45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:20:14,243 [236] [DEBUG] [app] Ending request: urn:request:a9a4679d-484b-4cb2-a540-7978e8f16c3e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9a4679d-484b-4cb2-a540-7978e8f16c3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:20:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:20:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:20:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:20:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:20:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:20:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:14,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:14,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:14,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:14,256 [235] [DEBUG] [app] Ending request: urn:request:f0d234e4-49f6-42eb-a421-a7cef7119abd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0d234e4-49f6-42eb-a421-a7cef7119abd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:14,256 [237] [DEBUG] [app] Ending request: urn:request:42511798-f97f-4302-b70d-1cb4f356bdcc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42511798-f97f-4302-b70d-1cb4f356bdcc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:20:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:20:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:20:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
logrotateworker stdout | 2025-02-07 05:20:14,722 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:20:16,115 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:20:16,892 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:20:16,949 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:20:17,603 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:20:17,606 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:20:17,609 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:20:17,611 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:20:17,615 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:20:18,211 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:18,518 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:20:18,627 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:20:18,631 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:20:18,634 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:20:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:20:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:28.732147+00:00 (in 9.999540 seconds)
notificationworker stdout | 2025-02-07 05:20:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:28 UTC)" (scheduled at 2025-02-07 05:20:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:20:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:20:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 18, 732883), True, datetime.datetime(2025, 2, 7, 5, 20, 18, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:20:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:20:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:20:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:20:18,947 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:20:19,026 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:20:19,292 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:20:19,640 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:20:20,008 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:20:20,012 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:20:20,014 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:20:20,017 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:20:20,021 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:20:20,024 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:20:20,027 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:20:20,030 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gcworker stdout | 2025-02-07 05:20:20,034 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
gunicorn-registry stdout | 2025-02-07 05:20:20,036 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
securityscanningnotificationworker stdout | 2025-02-07 05:20:20,258 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:20:20,508 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:20:21,900 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:20:22,107 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:20:22,473 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:20:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:20:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:03.419110+00:00 (in 39.997732 seconds)
repositorygcworker stdout | 2025-02-07 05:20:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:23 UTC)" (scheduled at 2025-02-07 05:20:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:20:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:20:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:20:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:20:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:40.424250+00:00 (in 16.000749 seconds)
gcworker stdout | 2025-02-07 05:20:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:54 UTC)" (scheduled at 2025-02-07 05:20:24.423055+00:00)
gcworker stdout | 2025-02-07 05:20:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:20:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:20:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:20:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:20:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:38.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 05:20:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:38 UTC)" (scheduled at 2025-02-07 05:20:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:20:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:20:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 28, 732862), True, datetime.datetime(2025, 2, 7, 5, 20, 28, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:20:28,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:20:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:20:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:20:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:20:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:44.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 05:20:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:44 UTC)" (scheduled at 2025-02-07 05:20:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:20:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:20:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:20:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:20:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:20:29,235 [237] [DEBUG] [app] Starting request: urn:request:c1e490a8-490f-4f26-b0b9-1573a818027b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:29,235 [236] [DEBUG] [app] Starting request: urn:request:3f54647b-fe64-4460-997a-9b46081632e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:20:29,238 [245] [DEBUG] [app] Starting request: urn:request:c47823e9-58a1-4075-8180-739845f46f17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:29,238 [246] [DEBUG] [app] Starting request: urn:request:a74a9835-e459-4034-afa3-a07b5e0964eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:29,238 [246] [DEBUG] [app] Ending request: urn:request:a74a9835-e459-4034-afa3-a07b5e0964eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a74a9835-e459-4034-afa3-a07b5e0964eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:20:29,238 [245] [DEBUG] [app] Ending request: urn:request:c47823e9-58a1-4075-8180-739845f46f17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c47823e9-58a1-4075-8180-739845f46f17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:20:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:20:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:29,241 [237] [DEBUG] [app] Starting request: urn:request:47355bb9-1088-4701-8055-cef3d760e843 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:20:29,241 [237] [DEBUG] [app] Ending request: urn:request:47355bb9-1088-4701-8055-cef3d760e843 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47355bb9-1088-4701-8055-cef3d760e843', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:20:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:29,242 [235] [DEBUG] [app] Starting request: urn:request:3bec823b-10fe-4cb9-b322-e4160d1621e3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:20:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:20:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:20:29,242 [235] [DEBUG] [app] Ending request: urn:request:3bec823b-10fe-4cb9-b322-e4160d1621e3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3bec823b-10fe-4cb9-b322-e4160d1621e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:20:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:20:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:20:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:20:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:29,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:29,253 [237] [DEBUG] [app] Ending request: urn:request:c1e490a8-490f-4f26-b0b9-1573a818027b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c1e490a8-490f-4f26-b0b9-1573a818027b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:20:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:20:29,254 [236] [DEBUG] [app] Ending request: urn:request:3f54647b-fe64-4460-997a-9b46081632e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f54647b-fe64-4460-997a-9b46081632e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:20:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
storagereplication stdout | 2025-02-07 05:20:31,443 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:20:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:20:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:32.625945+00:00 (in 1.004329 seconds)
namespacegcworker stdout | 2025-02-07 05:20:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:31 UTC)" (scheduled at 2025-02-07 05:20:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:20:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:20:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 31, 621815), True, datetime.datetime(2025, 2, 7, 5, 20, 31, 621815), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:20:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:20:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:20:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:20:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:20:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:31.621131+00:00 (in 58.994718 seconds)
namespacegcworker stdout | 2025-02-07 05:20:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:32 UTC)" (scheduled at 2025-02-07 05:20:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:20:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:20:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:20:33,217 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:20:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:20:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:03.919286+00:00 (in 29.999510 seconds)
buildlogsarchiver stdout | 2025-02-07 05:20:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:03 UTC)" (scheduled at 2025-02-07 05:20:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:20:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 20, 33, 920062), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:20:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:20:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:20:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:20:34,329 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:56.713547+00:00 (in 18.997205 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:37 UTC)" (scheduled at 2025-02-07 05:20:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:20:38,613 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:20:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:20:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:48.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 05:20:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:48 UTC)" (scheduled at 2025-02-07 05:20:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:20:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:20:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 38, 732875), True, datetime.datetime(2025, 2, 7, 5, 20, 38, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:20:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:20:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:20:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:20:39,625 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:20:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:20:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:54.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 05:20:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:10 UTC)" (scheduled at 2025-02-07 05:20:40.424250+00:00)
gcworker stdout | 2025-02-07 05:20:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:20:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905340431, None, 1, 0])
gcworker stdout | 2025-02-07 05:20:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:20:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:20:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:20:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:11.831521+00:00 (in 29.999532 seconds)
autopruneworker stdout | 2025-02-07 05:20:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:11 UTC)" (scheduled at 2025-02-07 05:20:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:20:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902041836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:20:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:20:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:20:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:20:43,559 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:20:44,126 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:20:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:20:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:59.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 05:20:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:59 UTC)" (scheduled at 2025-02-07 05:20:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:20:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:20:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:20:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:20:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:20:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:20:44,235 [235] [DEBUG] [app] Starting request: urn:request:60a3b441-a303-489c-8032-143ed6881aad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:44,235 [238] [DEBUG] [app] Starting request: urn:request:eef17f45-36d2-4e2a-988a-9e6e4757079b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:20:44,238 [239] [DEBUG] [app] Starting request: urn:request:897871d3-c6d8-440e-83c7-d8f07223d0f7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:44,238 [244] [DEBUG] [app] Starting request: urn:request:03f6de17-70f1-4404-b0a0-f0fabdb8ac3a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:44,238 [244] [DEBUG] [app] Ending request: urn:request:03f6de17-70f1-4404-b0a0-f0fabdb8ac3a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03f6de17-70f1-4404-b0a0-f0fabdb8ac3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:20:44,238 [239] [DEBUG] [app] Ending request: urn:request:897871d3-c6d8-440e-83c7-d8f07223d0f7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:897871d3-c6d8-440e-83c7-d8f07223d0f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:20:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:20:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:44,241 [237] [DEBUG] [app] Starting request: urn:request:ad1f9205-0f52-4184-b4eb-1fd4688ffd8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:20:44,242 [237] [DEBUG] [app] Ending request: urn:request:ad1f9205-0f52-4184-b4eb-1fd4688ffd8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad1f9205-0f52-4184-b4eb-1fd4688ffd8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:20:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:44,242 [236] [DEBUG] [app] Starting request: urn:request:afafe9af-ca12-4f15-be4b-24ded2c90283 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:20:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:44,242 [236] [DEBUG] [app] Ending request: urn:request:afafe9af-ca12-4f15-be4b-24ded2c90283 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:afafe9af-ca12-4f15-be4b-24ded2c90283', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:20:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:20:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:20:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:20:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:20:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:44,253 [235] [DEBUG] [app] Ending request: urn:request:60a3b441-a303-489c-8032-143ed6881aad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:60a3b441-a303-489c-8032-143ed6881aad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:20:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:20:44,253 [238] [DEBUG] [app] Ending request: urn:request:eef17f45-36d2-4e2a-988a-9e6e4757079b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eef17f45-36d2-4e2a-988a-9e6e4757079b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:20:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
logrotateworker stdout | 2025-02-07 05:20:44,758 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:20:46,141 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:20:46,918 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:20:46,982 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:20:47,611 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:20:47,614 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:20:47,617 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:20:47,620 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:20:47,623 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:20:48,226 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:48,554 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:20:48,635 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:20:48,638 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:20:48,641 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:20:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:20:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:50.814677+00:00 (in 2.082086 seconds)
notificationworker stdout | 2025-02-07 05:20:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:58 UTC)" (scheduled at 2025-02-07 05:20:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:20:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:20:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 48, 732857), True, datetime.datetime(2025, 2, 7, 5, 20, 48, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:20:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:20:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:20:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:20:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:20:48,980 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:20:49,042 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:20:49,318 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:20:49,677 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:20:50,018 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:20:50,021 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:20:50,023 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:20:50,027 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:20:50,032 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:20:50,034 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:20:50,038 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:20:50,042 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:20:50,045 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:20:50,069 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:20:50,294 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:20:50,521 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:20:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:20:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:20:58.732147+00:00 (in 7.917027 seconds)
notificationworker stdout | 2025-02-07 05:20:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:50 UTC)" (scheduled at 2025-02-07 05:20:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:20:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:20:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:20:51,936 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:20:52,130 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:20:52,493 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:20:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:20:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:10.424250+00:00 (in 16.000756 seconds)
gcworker stdout | 2025-02-07 05:20:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:24 UTC)" (scheduled at 2025-02-07 05:20:54.423055+00:00)
gcworker stdout | 2025-02-07 05:20:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:20:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:20:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:20:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:56.118855+00:00 (in 59.999475 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:20:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:56 UTC)" (scheduled at 2025-02-07 05:20:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:20:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:20:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:20:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:37.715878+00:00 (in 41.001890 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:56 UTC)" (scheduled at 2025-02-07 05:20:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:20:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:20:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:20:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:08.732147+00:00 (in 9.999379 seconds)
notificationworker stdout | 2025-02-07 05:20:58,733 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:08 UTC)" (scheduled at 2025-02-07 05:20:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:20:58,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:20:58,734 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 20, 58, 733249), True, datetime.datetime(2025, 2, 7, 5, 20, 58, 733249), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:20:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:20:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:20:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:20:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:20:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:14.225378+00:00 (in 14.999522 seconds)
quotatotalworker stdout | 2025-02-07 05:20:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:14 UTC)" (scheduled at 2025-02-07 05:20:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:20:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:20:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:20:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:20:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:20:59,235 [235] [DEBUG] [app] Starting request: urn:request:3544e647-db0d-4201-a839-7838c574963d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:59,235 [238] [DEBUG] [app] Starting request: urn:request:4c8095c2-666f-48eb-91d7-06e149a70e35 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:20:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:20:59,238 [246] [DEBUG] [app] Starting request: urn:request:74aab705-655b-4e88-9583-ad437a99af3a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:59,238 [239] [DEBUG] [app] Starting request: urn:request:e5bbd00c-6902-4ed8-9ec5-00dc0446448e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:20:59,238 [246] [DEBUG] [app] Ending request: urn:request:74aab705-655b-4e88-9583-ad437a99af3a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:74aab705-655b-4e88-9583-ad437a99af3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:20:59,238 [239] [DEBUG] [app] Ending request: urn:request:e5bbd00c-6902-4ed8-9ec5-00dc0446448e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5bbd00c-6902-4ed8-9ec5-00dc0446448e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:20:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:20:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:20:59,241 [238] [DEBUG] [app] Starting request: urn:request:a1017795-46db-4442-9945-8d5b9914b8cb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:20:59,241 [237] [DEBUG] [app] Starting request: urn:request:f0c64d0a-f456-4a61-b79b-b55a8670176b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:20:59,242 [238] [DEBUG] [app] Ending request: urn:request:a1017795-46db-4442-9945-8d5b9914b8cb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1017795-46db-4442-9945-8d5b9914b8cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:20:59,242 [237] [DEBUG] [app] Ending request: urn:request:f0c64d0a-f456-4a61-b79b-b55a8670176b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f0c64d0a-f456-4a61-b79b-b55a8670176b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:20:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:20:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:20:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:20:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:20:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:20:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:20:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:20:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:20:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:20:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:20:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:20:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:20:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:20:59,255 [235] [DEBUG] [app] Ending request: urn:request:3544e647-db0d-4201-a839-7838c574963d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3544e647-db0d-4201-a839-7838c574963d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:59,255 [238] [DEBUG] [app] Ending request: urn:request:4c8095c2-666f-48eb-91d7-06e149a70e35 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c8095c2-666f-48eb-91d7-06e149a70e35', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:20:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:20:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:20:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:20:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:20:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:20:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 05:20:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:20:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:04.936688+00:00 (in 4.998174 seconds)
exportactionlogsworker stdout | 2025-02-07 05:20:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:59 UTC)" (scheduled at 2025-02-07 05:20:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:20:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:20:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:21:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:21:01,479 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:21:03,253 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:21:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:21:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:23.420886+00:00 (in 20.001339 seconds)
repositorygcworker stdout | 2025-02-07 05:21:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:03 UTC)" (scheduled at 2025-02-07 05:21:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:21:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:21:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 3, 419813), True, datetime.datetime(2025, 2, 7, 5, 21, 3, 419813), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:21:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:21:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:21:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:21:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:21:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:33.919286+00:00 (in 29.999494 seconds)
buildlogsarchiver stdout | 2025-02-07 05:21:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:33 UTC)" (scheduled at 2025-02-07 05:21:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:21:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 21, 3, 920067), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:21:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:21:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:21:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:21:04,337 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:21:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:21:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:59.937899+00:00 (in 55.000746 seconds)
exportactionlogsworker stdout | 2025-02-07 05:21:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:04 UTC)" (scheduled at 2025-02-07 05:21:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:21:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:21:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 4, 937487), True, datetime.datetime(2025, 2, 7, 5, 21, 4, 937487), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:21:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:21:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:21:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:21:08,649 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:21:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:21:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:18.732147+00:00 (in 9.999423 seconds)
notificationworker stdout | 2025-02-07 05:21:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:18 UTC)" (scheduled at 2025-02-07 05:21:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:21:08,733 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:21:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 8, 733032), True, datetime.datetime(2025, 2, 7, 5, 21, 8, 733032), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:21:08,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:21:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:21:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:21:09,662 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:21:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:21:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:24.423055+00:00 (in 13.998381 seconds)
gcworker stdout | 2025-02-07 05:21:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:40 UTC)" (scheduled at 2025-02-07 05:21:10.424250+00:00)
gcworker stdout | 2025-02-07 05:21:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:21:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905370430, None, 1, 0])
gcworker stdout | 2025-02-07 05:21:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:21:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:21:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:21:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:41.831521+00:00 (in 29.999574 seconds)
autopruneworker stdout | 2025-02-07 05:21:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:41 UTC)" (scheduled at 2025-02-07 05:21:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:21:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902071836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:21:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:21:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:21:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:21:13,589 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:21:14,163 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:21:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:21:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:29.225378+00:00 (in 14.999535 seconds)
quotatotalworker stdout | 2025-02-07 05:21:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:29 UTC)" (scheduled at 2025-02-07 05:21:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:21:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:21:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:21:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:21:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:21:14,235 [235] [DEBUG] [app] Starting request: urn:request:7a9be674-a1a7-4165-adac-5bb8fe548c8a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:14,235 [238] [DEBUG] [app] Starting request: urn:request:cdc29ec1-474d-43ad-97a1-88f23f2d6d59 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:21:14,238 [246] [DEBUG] [app] Starting request: urn:request:77f17ed6-f4d3-4aaf-8e2e-1a5849e488ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:14,238 [239] [DEBUG] [app] Starting request: urn:request:da2c3aa9-7f4e-45ae-9206-c445972f9529 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:14,238 [246] [DEBUG] [app] Ending request: urn:request:77f17ed6-f4d3-4aaf-8e2e-1a5849e488ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77f17ed6-f4d3-4aaf-8e2e-1a5849e488ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:21:14,239 [239] [DEBUG] [app] Ending request: urn:request:da2c3aa9-7f4e-45ae-9206-c445972f9529 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da2c3aa9-7f4e-45ae-9206-c445972f9529', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:21:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:21:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:14,241 [235] [DEBUG] [app] Starting request: urn:request:38d1f8f9-b9dd-47e3-9439-3c3098f1969e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:14,241 [235] [DEBUG] [app] Ending request: urn:request:38d1f8f9-b9dd-47e3-9439-3c3098f1969e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:38d1f8f9-b9dd-47e3-9439-3c3098f1969e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:21:14,241 [237] [DEBUG] [app] Starting request: urn:request:d9eb4ed7-9ac4-4d9b-8116-227354aaa57d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:14,242 [237] [DEBUG] [app] Ending request: urn:request:d9eb4ed7-9ac4-4d9b-8116-227354aaa57d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9eb4ed7-9ac4-4d9b-8116-227354aaa57d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:21:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:21:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:21:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:14,253 [235] [DEBUG] [app] Ending request: urn:request:7a9be674-a1a7-4165-adac-5bb8fe548c8a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a9be674-a1a7-4165-adac-5bb8fe548c8a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:14,253 [238] [DEBUG] [app] Ending request: urn:request:cdc29ec1-474d-43ad-97a1-88f23f2d6d59 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cdc29ec1-474d-43ad-97a1-88f23f2d6d59', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:21:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:21:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:21:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
logrotateworker stdout | 2025-02-07 05:21:14,795 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:21:16,177 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:21:16,950 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:21:17,019 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:21:17,619 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:21:17,624 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:21:17,627 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:21:17,630 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:21:17,633 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:21:18,258 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:18,561 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:21:18,642 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:21:18,645 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:21:18,647 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:21:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:21:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:28.732147+00:00 (in 9.999488 seconds)
notificationworker stdout | 2025-02-07 05:21:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:28 UTC)" (scheduled at 2025-02-07 05:21:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:21:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:21:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 18, 732966), True, datetime.datetime(2025, 2, 7, 5, 21, 18, 732966), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:21:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:21:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:21:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:21:19,011 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:21:19,078 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:21:19,330 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:21:19,698 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:21:20,026 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:21:20,033 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:21:20,036 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:21:20,039 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:21:20,042 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:21:20,045 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:21:20,048 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:21:20,052 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:21:20,054 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:21:20,078 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:21:20,324 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:21:20,557 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:21:21,973 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:21:22,167 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:21:22,529 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:21:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:21:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:03.419110+00:00 (in 39.997702 seconds)
repositorygcworker stdout | 2025-02-07 05:21:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:23 UTC)" (scheduled at 2025-02-07 05:21:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:21:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:21:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:21:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:21:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:40.424250+00:00 (in 16.000708 seconds)
gcworker stdout | 2025-02-07 05:21:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:54 UTC)" (scheduled at 2025-02-07 05:21:24.423055+00:00)
gcworker stdout | 2025-02-07 05:21:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:21:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:21:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:21:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:21:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:38.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 05:21:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:38 UTC)" (scheduled at 2025-02-07 05:21:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:21:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:21:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 28, 732906), True, datetime.datetime(2025, 2, 7, 5, 21, 28, 732906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:21:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:21:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:21:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:21:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:21:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:44.225378+00:00 (in 14.999556 seconds)
quotatotalworker stdout | 2025-02-07 05:21:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:44 UTC)" (scheduled at 2025-02-07 05:21:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:21:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:21:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:21:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:21:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:21:29,236 [237] [DEBUG] [app] Starting request: urn:request:5305af07-1407-4511-8760-a980ea808497 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:29,236 [235] [DEBUG] [app] Starting request: urn:request:5ebcc832-210b-43ca-912a-1ec54841588b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:21:29,239 [246] [DEBUG] [app] Starting request: urn:request:e18bfea5-89a7-40c2-ad27-7bcb3cfffc73 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:29,239 [242] [DEBUG] [app] Starting request: urn:request:df842f6d-b20b-4f7d-9ac9-be78b77c3406 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:29,240 [246] [DEBUG] [app] Ending request: urn:request:e18bfea5-89a7-40c2-ad27-7bcb3cfffc73 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e18bfea5-89a7-40c2-ad27-7bcb3cfffc73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:21:29,240 [242] [DEBUG] [app] Ending request: urn:request:df842f6d-b20b-4f7d-9ac9-be78b77c3406 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df842f6d-b20b-4f7d-9ac9-be78b77c3406', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:21:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:21:29,240 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:29,243 [238] [DEBUG] [app] Starting request: urn:request:24b34ff7-9019-48d3-9c59-10d09d6f3f31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:29,243 [236] [DEBUG] [app] Starting request: urn:request:6a9fea05-994c-4104-8de4-0e19a74acd13 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:29,243 [238] [DEBUG] [app] Ending request: urn:request:24b34ff7-9019-48d3-9c59-10d09d6f3f31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:24b34ff7-9019-48d3-9c59-10d09d6f3f31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:21:29,243 [236] [DEBUG] [app] Ending request: urn:request:6a9fea05-994c-4104-8de4-0e19a74acd13 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a9fea05-994c-4104-8de4-0e19a74acd13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:21:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:21:29,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:21:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:21:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:29,255 [237] [DEBUG] [app] Ending request: urn:request:5305af07-1407-4511-8760-a980ea808497 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5305af07-1407-4511-8760-a980ea808497', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:29,255 [235] [DEBUG] [app] Ending request: urn:request:5ebcc832-210b-43ca-912a-1ec54841588b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ebcc832-210b-43ca-912a-1ec54841588b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:21:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:21:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:21:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
storagereplication stdout | 2025-02-07 05:21:31,515 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:21:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:21:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:32.625945+00:00 (in 1.004327 seconds)
namespacegcworker stdout | 2025-02-07 05:21:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:31 UTC)" (scheduled at 2025-02-07 05:21:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:21:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:21:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 31, 621828), True, datetime.datetime(2025, 2, 7, 5, 21, 31, 621828), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:21:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:21:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:21:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:21:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:21:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:13.624301+00:00 (in 40.997879 seconds)
namespacegcworker stdout | 2025-02-07 05:21:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:32 UTC)" (scheduled at 2025-02-07 05:21:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:21:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:21:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:21:33,282 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:21:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:21:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:03.919286+00:00 (in 29.999567 seconds)
buildlogsarchiver stdout | 2025-02-07 05:21:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:03 UTC)" (scheduled at 2025-02-07 05:21:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:21:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 21, 33, 919959), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:21:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:21:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:21:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:21:34,373 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:56.713547+00:00 (in 18.997185 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:37 UTC)" (scheduled at 2025-02-07 05:21:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:37 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:21:38,674 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:21:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:21:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:48.732147+00:00 (in 9.999553 seconds)
notificationworker stdout | 2025-02-07 05:21:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:48 UTC)" (scheduled at 2025-02-07 05:21:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:21:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:21:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 38, 732859), True, datetime.datetime(2025, 2, 7, 5, 21, 38, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:21:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:21:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:21:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:21:39,699 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:21:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:21:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:54.423055+00:00 (in 13.998395 seconds)
gcworker stdout | 2025-02-07 05:21:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:10 UTC)" (scheduled at 2025-02-07 05:21:40.424250+00:00)
gcworker stdout | 2025-02-07 05:21:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:21:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905400431, None, 1, 0])
gcworker stdout | 2025-02-07 05:21:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:21:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:21:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:21:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:11.831521+00:00 (in 29.999540 seconds)
autopruneworker stdout | 2025-02-07 05:21:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:11 UTC)" (scheduled at 2025-02-07 05:21:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:21:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902101836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:21:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:21:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:21:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:21:43,599 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:21:44,199 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:21:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:21:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:59.225378+00:00 (in 14.999514 seconds)
quotatotalworker stdout | 2025-02-07 05:21:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:59 UTC)" (scheduled at 2025-02-07 05:21:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:21:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:21:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:21:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:21:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:21:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:21:44,235 [235] [DEBUG] [app] Starting request: urn:request:54c4e31c-e6d8-4460-a72c-402e8483f912 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:44,235 [238] [DEBUG] [app] Starting request: urn:request:cf0b1219-b887-4f0e-9212-1126abe6bad3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:21:44,238 [239] [DEBUG] [app] Starting request: urn:request:20ce9c88-7afe-47b0-9673-2fbc088c742c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:44,238 [246] [DEBUG] [app] Starting request: urn:request:f9357cf2-ce29-4853-9bd1-453e0335874d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:44,239 [239] [DEBUG] [app] Ending request: urn:request:20ce9c88-7afe-47b0-9673-2fbc088c742c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20ce9c88-7afe-47b0-9673-2fbc088c742c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:21:44,239 [246] [DEBUG] [app] Ending request: urn:request:f9357cf2-ce29-4853-9bd1-453e0335874d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f9357cf2-ce29-4853-9bd1-453e0335874d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:21:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:21:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:21:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:44,242 [238] [DEBUG] [app] Starting request: urn:request:441333f9-1b78-4f41-94f1-c1a6b6788581 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:44,242 [235] [DEBUG] [app] Starting request: urn:request:66941aa5-1d60-46a3-9b77-f384d111fa1c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:44,242 [235] [DEBUG] [app] Ending request: urn:request:66941aa5-1d60-46a3-9b77-f384d111fa1c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66941aa5-1d60-46a3-9b77-f384d111fa1c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:21:44,242 [238] [DEBUG] [app] Ending request: urn:request:441333f9-1b78-4f41-94f1-c1a6b6788581 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:441333f9-1b78-4f41-94f1-c1a6b6788581', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:21:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:21:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:21:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:21:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:44,254 [238] [DEBUG] [app] Ending request: urn:request:cf0b1219-b887-4f0e-9212-1126abe6bad3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf0b1219-b887-4f0e-9212-1126abe6bad3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:44,254 [235] [DEBUG] [app] Ending request: urn:request:54c4e31c-e6d8-4460-a72c-402e8483f912 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:54c4e31c-e6d8-4460-a72c-402e8483f912', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:21:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:21:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
gunicorn-web stdout | 2025-02-07 05:21:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
logrotateworker stdout | 2025-02-07 05:21:44,831 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:21:46,208 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:21:46,979 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:21:47,055 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:21:47,627 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:21:47,631 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:21:47,635 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:21:47,639 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:21:47,642 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:21:48,272 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:48,597 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:21:48,650 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:21:48,653 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:21:48,655 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:21:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:21:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:50.814677+00:00 (in 2.082094 seconds)
notificationworker stdout | 2025-02-07 05:21:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:58 UTC)" (scheduled at 2025-02-07 05:21:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:21:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:21:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 48, 732851), True, datetime.datetime(2025, 2, 7, 5, 21, 48, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:21:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:21:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:21:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:21:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:21:49,048 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:21:49,115 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:21:49,366 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:21:49,707 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:21:50,034 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:21:50,044 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:21:50,048 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:21:50,052 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:21:50,055 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:21:50,058 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:21:50,060 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:21:50,064 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:21:50,067 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:21:50,115 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:21:50,361 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:21:50,578 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:21:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:21:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:21:58.732147+00:00 (in 7.917017 seconds)
notificationworker stdout | 2025-02-07 05:21:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:50 UTC)" (scheduled at 2025-02-07 05:21:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:21:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:21:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:21:52,009 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:21:52,199 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:21:52,565 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:21:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:21:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:10.424250+00:00 (in 16.000762 seconds)
gcworker stdout | 2025-02-07 05:21:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:24 UTC)" (scheduled at 2025-02-07 05:21:54.423055+00:00)
gcworker stdout | 2025-02-07 05:21:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:21:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:21:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:21:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:56.118855+00:00 (in 59.999473 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:21:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:56 UTC)" (scheduled at 2025-02-07 05:21:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:21:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:21:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:21:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:37.715878+00:00 (in 41.001828 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:56 UTC)" (scheduled at 2025-02-07 05:21:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:21:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:21:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:21:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:08.732147+00:00 (in 9.999581 seconds)
notificationworker stdout | 2025-02-07 05:21:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:08 UTC)" (scheduled at 2025-02-07 05:21:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:21:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:21:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 21, 58, 732876), True, datetime.datetime(2025, 2, 7, 5, 21, 58, 732876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:21:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:21:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:21:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:21:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:21:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:14.225378+00:00 (in 14.999519 seconds)
quotatotalworker stdout | 2025-02-07 05:21:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:14 UTC)" (scheduled at 2025-02-07 05:21:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:21:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:21:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:21:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:21:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:21:59,235 [238] [DEBUG] [app] Starting request: urn:request:2fc06653-1d72-4e48-b88f-8565a99731e1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:59,235 [237] [DEBUG] [app] Starting request: urn:request:dc509ea5-51d3-490b-8a6b-0b29c04ac054 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:21:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:21:59,238 [239] [DEBUG] [app] Starting request: urn:request:2344ff9e-fda8-4d24-bcf2-5115046b597f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:59,238 [246] [DEBUG] [app] Starting request: urn:request:7cb60b74-ddfa-45b9-a5b5-119e8313603a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:21:59,239 [239] [DEBUG] [app] Ending request: urn:request:2344ff9e-fda8-4d24-bcf2-5115046b597f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2344ff9e-fda8-4d24-bcf2-5115046b597f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:21:59,239 [246] [DEBUG] [app] Ending request: urn:request:7cb60b74-ddfa-45b9-a5b5-119e8313603a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7cb60b74-ddfa-45b9-a5b5-119e8313603a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:21:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:21:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:21:59,242 [237] [DEBUG] [app] Starting request: urn:request:4354a796-bdf2-416e-8564-ad03d85a65ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:59,242 [237] [DEBUG] [app] Ending request: urn:request:4354a796-bdf2-416e-8564-ad03d85a65ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4354a796-bdf2-416e-8564-ad03d85a65ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:21:59,243 [236] [DEBUG] [app] Starting request: urn:request:1849be36-df17-4d32-b556-d09905e88d18 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:21:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:21:59,243 [236] [DEBUG] [app] Ending request: urn:request:1849be36-df17-4d32-b556-d09905e88d18 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1849be36-df17-4d32-b556-d09905e88d18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:21:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:21:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:21:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:21:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:21:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:21:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:21:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:21:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:21:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:21:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:21:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:21:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:21:59,254 [237] [DEBUG] [app] Ending request: urn:request:dc509ea5-51d3-490b-8a6b-0b29c04ac054 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc509ea5-51d3-490b-8a6b-0b29c04ac054', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:21:59,255 [238] [DEBUG] [app] Ending request: urn:request:2fc06653-1d72-4e48-b88f-8565a99731e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2fc06653-1d72-4e48-b88f-8565a99731e1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:21:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:21:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:21:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:21:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:21:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:21:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:21:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:04.936688+00:00 (in 4.998301 seconds)
exportactionlogsworker stdout | 2025-02-07 05:21:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:59 UTC)" (scheduled at 2025-02-07 05:21:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:21:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:21:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:22:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:22:01,552 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:22:03,318 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:22:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:22:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:23.420886+00:00 (in 20.001361 seconds)
repositorygcworker stdout | 2025-02-07 05:22:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:03 UTC)" (scheduled at 2025-02-07 05:22:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:22:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:22:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 3, 419836), True, datetime.datetime(2025, 2, 7, 5, 22, 3, 419836), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:22:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:22:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:22:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:22:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:22:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:33.919286+00:00 (in 29.999548 seconds)
buildlogsarchiver stdout | 2025-02-07 05:22:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:33 UTC)" (scheduled at 2025-02-07 05:22:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:22:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 22, 3, 919989), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:22:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:22:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:22:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:22:04,411 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:22:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:22:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:59.937899+00:00 (in 55.000760 seconds)
exportactionlogsworker stdout | 2025-02-07 05:22:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:04 UTC)" (scheduled at 2025-02-07 05:22:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:22:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:22:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 4, 937451), True, datetime.datetime(2025, 2, 7, 5, 22, 4, 937451), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:22:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:22:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:22:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:04 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:22:08,710 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:22:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:18.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:22:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:18 UTC)" (scheduled at 2025-02-07 05:22:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:22:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:22:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 8, 732849), True, datetime.datetime(2025, 2, 7, 5, 22, 8, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:22:08,743 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:22:08,743 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:22:08,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:18 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:22:09,733 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:22:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:22:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:24.423055+00:00 (in 13.998346 seconds)
gcworker stdout | 2025-02-07 05:22:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:40 UTC)" (scheduled at 2025-02-07 05:22:10.424250+00:00)
gcworker stdout | 2025-02-07 05:22:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:22:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905430431, None, 1, 0])
gcworker stdout | 2025-02-07 05:22:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:22:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:22:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:22:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:41.831521+00:00 (in 29.999557 seconds)
autopruneworker stdout | 2025-02-07 05:22:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:41 UTC)" (scheduled at 2025-02-07 05:22:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:22:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902131836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:22:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:22:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:22:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:41 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:22:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:22:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:31.621131+00:00 (in 17.996383 seconds)
namespacegcworker stdout | 2025-02-07 05:22:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:27:13 UTC)" (scheduled at 2025-02-07 05:22:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:22:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 22, 13, 625036), 'namespacegc/%'])
quotatotalworker stdout | 2025-02-07 05:22:13,629 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
namespacegcworker stdout | 2025-02-07 05:22:13,632 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 22, 13, 625036), True, datetime.datetime(2025, 2, 7, 5, 22, 13, 625036), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:22:13,634 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 22, 13, 625036), True, datetime.datetime(2025, 2, 7, 5, 22, 13, 625036), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 22, 13, 625036), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:22:13,636 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:22:13,636 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:27:13 UTC)" executed successfully
builder stdout | 2025-02-07 05:22:14,211 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:22:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:22:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:29.225378+00:00 (in 14.999522 seconds)
quotatotalworker stdout | 2025-02-07 05:22:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:29 UTC)" (scheduled at 2025-02-07 05:22:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:22:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:22:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:22:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:22:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:22:14,236 [235] [DEBUG] [app] Starting request: urn:request:0ff85360-e37a-469e-82de-c10728c9aa1e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:14,236 [237] [DEBUG] [app] Starting request: urn:request:78ef6fa3-e8fe-49a8-8254-6f3a4eb2d80c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:22:14,239 [246] [DEBUG] [app] Starting request: urn:request:59edb560-437d-4cf8-a853-853835b749ef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:14,239 [245] [DEBUG] [app] Starting request: urn:request:583014de-152d-439f-9199-568a6c6ce584 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:14,239 [245] [DEBUG] [app] Ending request: urn:request:583014de-152d-439f-9199-568a6c6ce584 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:583014de-152d-439f-9199-568a6c6ce584', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:22:14,239 [246] [DEBUG] [app] Ending request: urn:request:59edb560-437d-4cf8-a853-853835b749ef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:59edb560-437d-4cf8-a853-853835b749ef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:22:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:22:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:22:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:14,242 [237] [DEBUG] [app] Starting request: urn:request:18a76e0a-733f-4fc4-b925-01df99edb2c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:14,243 [237] [DEBUG] [app] Ending request: urn:request:18a76e0a-733f-4fc4-b925-01df99edb2c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:18a76e0a-733f-4fc4-b925-01df99edb2c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:22:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:22:14,243 [238] [DEBUG] [app] Starting request: urn:request:d7fc8fa2-f480-455f-bcd7-26140c22329d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:14,243 [238] [DEBUG] [app] Ending request: urn:request:d7fc8fa2-f480-455f-bcd7-26140c22329d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d7fc8fa2-f480-455f-bcd7-26140c22329d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:22:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:22:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:22:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:14,254 [237] [DEBUG] [app] Ending request: urn:request:78ef6fa3-e8fe-49a8-8254-6f3a4eb2d80c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78ef6fa3-e8fe-49a8-8254-6f3a4eb2d80c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:22:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:22:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:22:14,255 [235] [DEBUG] [app] Ending request: urn:request:0ff85360-e37a-469e-82de-c10728c9aa1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ff85360-e37a-469e-82de-c10728c9aa1e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:22:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:22:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
logrotateworker stdout | 2025-02-07 05:22:14,855 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:22:16,234 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:22:17,012 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:22:17,078 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:22:17,635 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:22:17,638 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:22:17,641 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:22:17,646 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:22:17,648 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:22:18,308 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:18,633 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:22:18,658 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:22:18,661 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:22:18,664 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:22:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 05:22:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:28 UTC)" (scheduled at 2025-02-07 05:22:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:22:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:22:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 18, 732871), True, datetime.datetime(2025, 2, 7, 5, 22, 18, 732871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:22:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:22:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:22:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:22:19,085 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:22:19,148 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:22:19,374 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:22:19,718 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:22:20,047 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:22:20,054 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:22:20,057 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:22:20,060 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:22:20,063 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:22:20,066 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:22:20,068 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:22:20,072 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:22:20,075 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:22:20,146 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:22:20,397 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:22:20,602 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:22:22,037 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:22:22,212 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:22:22,578 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:22:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:22:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:03.419110+00:00 (in 39.997759 seconds)
repositorygcworker stdout | 2025-02-07 05:22:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:23 UTC)" (scheduled at 2025-02-07 05:22:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:22:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:22:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:22:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:22:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:40.424250+00:00 (in 16.000755 seconds)
gcworker stdout | 2025-02-07 05:22:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:54 UTC)" (scheduled at 2025-02-07 05:22:24.423055+00:00)
gcworker stdout | 2025-02-07 05:22:24,423 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 05:22:24,430 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:22:24,430 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:22:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:22:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:22:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:33.814166+00:00 (in 5.081583 seconds)
notificationworker stdout | 2025-02-07 05:22:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:38 UTC)" (scheduled at 2025-02-07 05:22:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:22:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:22:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 28, 732867), True, datetime.datetime(2025, 2, 7, 5, 22, 28, 732867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:22:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:22:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:22:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:22:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:22:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:44.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 05:22:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:44 UTC)" (scheduled at 2025-02-07 05:22:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:22:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:22:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:22:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:22:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:22:29,236 [238] [DEBUG] [app] Starting request: urn:request:96947faa-2740-44b0-a3c1-40f64f622e96 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:29,236 [235] [DEBUG] [app] Starting request: urn:request:6c91b81d-92f5-4948-b61e-c470ff847de7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:22:29,240 [246] [DEBUG] [app] Starting request: urn:request:17243ca2-bedc-49b3-9238-4f7cedea958c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:29,240 [239] [DEBUG] [app] Starting request: urn:request:123e53f8-165d-4495-a10a-acbfa9a02440 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:29,240 [246] [DEBUG] [app] Ending request: urn:request:17243ca2-bedc-49b3-9238-4f7cedea958c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:17243ca2-bedc-49b3-9238-4f7cedea958c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:22:29,240 [239] [DEBUG] [app] Ending request: urn:request:123e53f8-165d-4495-a10a-acbfa9a02440 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:123e53f8-165d-4495-a10a-acbfa9a02440', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:22:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:22:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:22:29,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:29,243 [238] [DEBUG] [app] Starting request: urn:request:b984cc95-c228-4b09-b3d1-ab8d97a0a80e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:29,243 [238] [DEBUG] [app] Ending request: urn:request:b984cc95-c228-4b09-b3d1-ab8d97a0a80e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b984cc95-c228-4b09-b3d1-ab8d97a0a80e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:22:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:22:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:29,244 [237] [DEBUG] [app] Starting request: urn:request:1820398d-b5b5-4a20-b929-b1712c14dbac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:29,244 [237] [DEBUG] [app] Ending request: urn:request:1820398d-b5b5-4a20-b929-b1712c14dbac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1820398d-b5b5-4a20-b929-b1712c14dbac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:22:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:22:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:29,255 [238] [DEBUG] [app] Ending request: urn:request:96947faa-2740-44b0-a3c1-40f64f622e96 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:96947faa-2740-44b0-a3c1-40f64f622e96', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:22:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:22:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:22:29,256 [235] [DEBUG] [app] Ending request: urn:request:6c91b81d-92f5-4948-b61e-c470ff847de7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c91b81d-92f5-4948-b61e-c470ff847de7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:22:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:22:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
storagereplication stdout | 2025-02-07 05:22:31,579 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:22:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:22:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:32.625945+00:00 (in 1.004383 seconds)
namespacegcworker stdout | 2025-02-07 05:22:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:31 UTC)" (scheduled at 2025-02-07 05:22:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:22:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:22:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 31, 621830), True, datetime.datetime(2025, 2, 7, 5, 22, 31, 621830), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:22:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:22:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:22:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:22:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:22:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:31.621131+00:00 (in 58.994746 seconds)
namespacegcworker stdout | 2025-02-07 05:22:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:32 UTC)" (scheduled at 2025-02-07 05:22:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:22:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:22:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:22:33,343 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 05:22:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:38.732147+00:00 (in 4.917519 seconds)
notificationworker stdout | 2025-02-07 05:22:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:27:33 UTC)" (scheduled at 2025-02-07 05:22:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:22:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 22, 33, 814880), 'notification/%'])
notificationworker stdout | 2025-02-07 05:22:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 22, 33, 814880), True, datetime.datetime(2025, 2, 7, 5, 22, 33, 814880), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:22:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 22, 33, 814880), True, datetime.datetime(2025, 2, 7, 5, 22, 33, 814880), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 22, 33, 814880), 'notification/%'])
notificationworker stdout | 2025-02-07 05:22:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:22:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:27:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:22:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:22:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:03.919286+00:00 (in 29.999558 seconds)
buildlogsarchiver stdout | 2025-02-07 05:22:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:03 UTC)" (scheduled at 2025-02-07 05:22:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:22:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 22, 33, 919978), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:22:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:22:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:22:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:22:34,436 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:56.713547+00:00 (in 18.997179 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:37 UTC)" (scheduled at 2025-02-07 05:22:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:22:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:48.732147+00:00 (in 9.999573 seconds)
notificationworker stdout | 2025-02-07 05:22:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:48 UTC)" (scheduled at 2025-02-07 05:22:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:22:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:22:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 38, 732772), True, datetime.datetime(2025, 2, 7, 5, 22, 38, 732772), 0, 'notification/%', 50, 1, 0])
teamsyncworker stdout | 2025-02-07 05:22:38,735 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2025-02-07 05:22:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:22:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:22:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:48 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:22:39,741 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:22:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:22:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:54.423055+00:00 (in 13.998345 seconds)
gcworker stdout | 2025-02-07 05:22:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:10 UTC)" (scheduled at 2025-02-07 05:22:40.424250+00:00)
gcworker stdout | 2025-02-07 05:22:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:22:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905460430, None, 1, 0])
gcworker stdout | 2025-02-07 05:22:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:22:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:22:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:22:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:11.831521+00:00 (in 29.999566 seconds)
autopruneworker stdout | 2025-02-07 05:22:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:11 UTC)" (scheduled at 2025-02-07 05:22:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:22:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902161836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:22:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:22:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:22:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:22:43,666 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:22:44,223 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
quotatotalworker stdout | 2025-02-07 05:22:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:22:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:59.225378+00:00 (in 14.999594 seconds)
quotatotalworker stdout | 2025-02-07 05:22:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:59 UTC)" (scheduled at 2025-02-07 05:22:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:22:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:22:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:22:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:22:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:22:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:22:44,235 [236] [DEBUG] [app] Starting request: urn:request:54c2a005-86ab-40ef-817f-8b2dd50c5f20 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:44,235 [237] [DEBUG] [app] Starting request: urn:request:c5858629-64fd-4197-8626-b03769f9a72e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:22:44,238 [239] [DEBUG] [app] Starting request: urn:request:21c1070f-dd79-42ba-b515-decdd7a116cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:44,238 [245] [DEBUG] [app] Starting request: urn:request:b5bb5bff-c47e-427b-b158-8106fa224998 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:44,239 [245] [DEBUG] [app] Ending request: urn:request:b5bb5bff-c47e-427b-b158-8106fa224998 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b5bb5bff-c47e-427b-b158-8106fa224998', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:22:44,239 [239] [DEBUG] [app] Ending request: urn:request:21c1070f-dd79-42ba-b515-decdd7a116cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:21c1070f-dd79-42ba-b515-decdd7a116cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:22:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:22:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:22:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:22:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:44,241 [238] [DEBUG] [app] Starting request: urn:request:4283b6d7-9e36-4cdb-a6c4-a1dc73b2a5c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:44,241 [237] [DEBUG] [app] Starting request: urn:request:3242432e-2253-44a4-ae82-54878862d1b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:44,242 [238] [DEBUG] [app] Ending request: urn:request:4283b6d7-9e36-4cdb-a6c4-a1dc73b2a5c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4283b6d7-9e36-4cdb-a6c4-a1dc73b2a5c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:22:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:22:44,242 [237] [DEBUG] [app] Ending request: urn:request:3242432e-2253-44a4-ae82-54878862d1b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3242432e-2253-44a4-ae82-54878862d1b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:22:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:22:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:44,253 [237] [DEBUG] [app] Ending request: urn:request:c5858629-64fd-4197-8626-b03769f9a72e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c5858629-64fd-4197-8626-b03769f9a72e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:22:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:22:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:22:44,254 [236] [DEBUG] [app] Ending request: urn:request:54c2a005-86ab-40ef-817f-8b2dd50c5f20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:54c2a005-86ab-40ef-817f-8b2dd50c5f20', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:22:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 05:22:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
logrotateworker stdout | 2025-02-07 05:22:44,891 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:22:46,270 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:22:47,036 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:22:47,109 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:22:47,644 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:22:47,648 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:22:47,650 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:22:47,653 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:22:47,656 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:22:48,343 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:48,640 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:22:48,665 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:22:48,668 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:22:48,670 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:22:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:50.814677+00:00 (in 2.082059 seconds)
notificationworker stdout | 2025-02-07 05:22:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:58 UTC)" (scheduled at 2025-02-07 05:22:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:22:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:22:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 48, 732823), True, datetime.datetime(2025, 2, 7, 5, 22, 48, 732823), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:22:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:22:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:22:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:22:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:22:49,121 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:22:49,184 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:22:49,410 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:22:49,741 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:22:50,058 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:22:50,062 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:22:50,067 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:22:50,070 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:22:50,073 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:22:50,075 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:22:50,079 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:22:50,082 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:22:50,086 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:22:50,170 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:22:50,434 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:22:50,618 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:22:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:22:58.732147+00:00 (in 7.917001 seconds)
notificationworker stdout | 2025-02-07 05:22:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:50 UTC)" (scheduled at 2025-02-07 05:22:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:22:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:22:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:22:52,057 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:22:52,219 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:22:52,615 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:22:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:22:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:10.424250+00:00 (in 16.000754 seconds)
gcworker stdout | 2025-02-07 05:22:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:24 UTC)" (scheduled at 2025-02-07 05:22:54.423055+00:00)
gcworker stdout | 2025-02-07 05:22:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:22:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:22:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:22:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:56.118855+00:00 (in 59.999533 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:22:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:56 UTC)" (scheduled at 2025-02-07 05:22:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:22:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:22:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:22:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:37.715878+00:00 (in 41.001865 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:56 UTC)" (scheduled at 2025-02-07 05:22:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:22:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:22:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:22:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:08.732147+00:00 (in 9.999547 seconds)
notificationworker stdout | 2025-02-07 05:22:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:08 UTC)" (scheduled at 2025-02-07 05:22:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:22:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:22:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 22, 58, 732873), True, datetime.datetime(2025, 2, 7, 5, 22, 58, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:22:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:22:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:22:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:22:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:22:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:14.225378+00:00 (in 14.999585 seconds)
quotatotalworker stdout | 2025-02-07 05:22:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:14 UTC)" (scheduled at 2025-02-07 05:22:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:22:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:22:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:22:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:22:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:22:59,235 [238] [DEBUG] [app] Starting request: urn:request:480244f7-2167-4483-b594-1ee00d03fc95 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:59,235 [235] [DEBUG] [app] Starting request: urn:request:ce410f81-095c-4bf0-9b61-05b7a4229731 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:22:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:22:59,238 [246] [DEBUG] [app] Starting request: urn:request:7e423d18-5992-4126-8d71-ea05af91c565 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:59,238 [245] [DEBUG] [app] Starting request: urn:request:25379386-30d1-4beb-83f1-e3090829c77d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:22:59,239 [246] [DEBUG] [app] Ending request: urn:request:7e423d18-5992-4126-8d71-ea05af91c565 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7e423d18-5992-4126-8d71-ea05af91c565', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:22:59,239 [245] [DEBUG] [app] Ending request: urn:request:25379386-30d1-4beb-83f1-e3090829c77d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:25379386-30d1-4beb-83f1-e3090829c77d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:22:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:22:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:22:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:22:59,241 [238] [DEBUG] [app] Starting request: urn:request:fefc1f55-da99-4d74-8208-44850a2fb1c5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:59,242 [237] [DEBUG] [app] Starting request: urn:request:16bc738d-0212-46c7-970c-45eed7dda92b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:22:59,242 [238] [DEBUG] [app] Ending request: urn:request:fefc1f55-da99-4d74-8208-44850a2fb1c5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fefc1f55-da99-4d74-8208-44850a2fb1c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:22:59,242 [237] [DEBUG] [app] Ending request: urn:request:16bc738d-0212-46c7-970c-45eed7dda92b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16bc738d-0212-46c7-970c-45eed7dda92b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:22:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:22:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:22:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:22:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:22:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:22:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:22:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:22:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:22:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:22:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:22:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:22:59,253 [238] [DEBUG] [app] Ending request: urn:request:480244f7-2167-4483-b594-1ee00d03fc95 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:480244f7-2167-4483-b594-1ee00d03fc95', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:22:59,253 [235] [DEBUG] [app] Ending request: urn:request:ce410f81-095c-4bf0-9b61-05b7a4229731 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce410f81-095c-4bf0-9b61-05b7a4229731', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:22:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:22:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:22:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:22:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:22:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:22:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
exportactionlogsworker stdout | 2025-02-07 05:22:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:22:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:04.936688+00:00 (in 4.998278 seconds)
exportactionlogsworker stdout | 2025-02-07 05:22:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:59 UTC)" (scheduled at 2025-02-07 05:22:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:22:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:22:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:23:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:23:01,589 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:23:03,379 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:23:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:23:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:23.420886+00:00 (in 20.001318 seconds)
repositorygcworker stdout | 2025-02-07 05:23:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:03 UTC)" (scheduled at 2025-02-07 05:23:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:23:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:23:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 3, 419834), True, datetime.datetime(2025, 2, 7, 5, 23, 3, 419834), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:23:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:23:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:23:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:23:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:23:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:33.919286+00:00 (in 29.999520 seconds)
buildlogsarchiver stdout | 2025-02-07 05:23:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:33 UTC)" (scheduled at 2025-02-07 05:23:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:23:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 23, 3, 920053), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:23:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:23:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:23:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:23:04,458 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:23:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:23:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:59.937899+00:00 (in 55.000781 seconds)
exportactionlogsworker stdout | 2025-02-07 05:23:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:04 UTC)" (scheduled at 2025-02-07 05:23:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:23:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:23:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 4, 937457), True, datetime.datetime(2025, 2, 7, 5, 23, 4, 937457), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:23:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:23:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:23:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:23:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:23:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:18.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 05:23:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:18 UTC)" (scheduled at 2025-02-07 05:23:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:23:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:23:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 8, 732910), True, datetime.datetime(2025, 2, 7, 5, 23, 8, 732910), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:23:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:23:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:23:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:23:08,772 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:23:09,764 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:23:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:23:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:24.423055+00:00 (in 13.998356 seconds)
gcworker stdout | 2025-02-07 05:23:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:40 UTC)" (scheduled at 2025-02-07 05:23:10.424250+00:00)
gcworker stdout | 2025-02-07 05:23:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:23:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905490431, None, 1, 0])
gcworker stdout | 2025-02-07 05:23:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:23:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:23:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:23:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:41.831521+00:00 (in 29.999569 seconds)
autopruneworker stdout | 2025-02-07 05:23:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:41 UTC)" (scheduled at 2025-02-07 05:23:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:23:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902191836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:23:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:23:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:23:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:23:13,702 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:23:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:23:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:29.225378+00:00 (in 14.999585 seconds)
quotatotalworker stdout | 2025-02-07 05:23:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:29 UTC)" (scheduled at 2025-02-07 05:23:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:23:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:23:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:23:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:23:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:23:14,236 [236] [DEBUG] [app] Starting request: urn:request:931aeae8-26c5-409b-bab6-1f45023d5d6d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:14,236 [237] [DEBUG] [app] Starting request: urn:request:9f2be2df-f858-4134-8891-1d5f414e1898 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:23:14,238 [246] [DEBUG] [app] Starting request: urn:request:71875971-c031-4933-8672-edbfd09680c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:14,238 [239] [DEBUG] [app] Starting request: urn:request:cbbdc755-c820-4c5a-acda-ac3846069171 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:14,239 [246] [DEBUG] [app] Ending request: urn:request:71875971-c031-4933-8672-edbfd09680c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71875971-c031-4933-8672-edbfd09680c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:23:14,239 [239] [DEBUG] [app] Ending request: urn:request:cbbdc755-c820-4c5a-acda-ac3846069171 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cbbdc755-c820-4c5a-acda-ac3846069171', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:23:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:23:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:14,242 [235] [DEBUG] [app] Starting request: urn:request:fd950322-23d3-4461-8752-1ff3fedd9d85 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:14,242 [238] [DEBUG] [app] Starting request: urn:request:468cac8b-21e4-437e-af8b-f23a7eb52367 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:14,243 [235] [DEBUG] [app] Ending request: urn:request:fd950322-23d3-4461-8752-1ff3fedd9d85 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd950322-23d3-4461-8752-1ff3fedd9d85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:23:14,243 [238] [DEBUG] [app] Ending request: urn:request:468cac8b-21e4-437e-af8b-f23a7eb52367 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:468cac8b-21e4-437e-af8b-f23a7eb52367', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:23:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:23:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:23:14,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:23:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:23:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:23:14,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:14,255 [237] [DEBUG] [app] Ending request: urn:request:9f2be2df-f858-4134-8891-1d5f414e1898 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f2be2df-f858-4134-8891-1d5f414e1898', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:14,255 [236] [DEBUG] [app] Ending request: urn:request:931aeae8-26c5-409b-bab6-1f45023d5d6d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:931aeae8-26c5-409b-bab6-1f45023d5d6d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:23:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
builder stdout | 2025-02-07 05:23:14,259 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:23:14,918 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:23:16,306 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:23:17,057 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:23:17,131 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:23:17,651 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:23:17,654 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:23:17,656 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:23:17,659 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:23:17,662 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:23:18,379 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:23:18,674 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:23:18,677 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:18,680 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-secscan stdout | 2025-02-07 05:23:18,682 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:23:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:23:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:28.732147+00:00 (in 9.999573 seconds)
notificationworker stdout | 2025-02-07 05:23:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:28 UTC)" (scheduled at 2025-02-07 05:23:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:23:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:23:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 18, 732883), True, datetime.datetime(2025, 2, 7, 5, 23, 18, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:23:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:23:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:23:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:28 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:23:19,157 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:23:19,217 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:23:19,437 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:23:19,774 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:23:20,068 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:23:20,071 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:23:20,076 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:23:20,080 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:23:20,083 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:23:20,085 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:23:20,088 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:23:20,090 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:23:20,095 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:23:20,181 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:23:20,470 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:23:20,651 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:23:22,094 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:23:22,234 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:23:22,622 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:23:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:23:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:32.420707+00:00 (in 8.999348 seconds)
repositorygcworker stdout | 2025-02-07 05:23:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:23 UTC)" (scheduled at 2025-02-07 05:23:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:23:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:23:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:23:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:23:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 05:23:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:54 UTC)" (scheduled at 2025-02-07 05:23:24.423055+00:00)
gcworker stdout | 2025-02-07 05:23:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:23:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:23:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:23:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:23:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:38.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 05:23:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:38 UTC)" (scheduled at 2025-02-07 05:23:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:23:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:23:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 28, 732795), True, datetime.datetime(2025, 2, 7, 5, 23, 28, 732795), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:23:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:23:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:23:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:23:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:23:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:44.225378+00:00 (in 14.999571 seconds)
quotatotalworker stdout | 2025-02-07 05:23:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:44 UTC)" (scheduled at 2025-02-07 05:23:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:23:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:23:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:23:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:23:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:23:29,235 [236] [DEBUG] [app] Starting request: urn:request:c49ab8a3-fa48-4688-999a-87a1bbc4f0b9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:29,235 [237] [DEBUG] [app] Starting request: urn:request:77d0fb98-280a-42c2-8550-c19d27f0c169 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:23:29,238 [246] [DEBUG] [app] Starting request: urn:request:9a3c7010-50ed-4dde-9f34-64f41160aa13 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:29,238 [245] [DEBUG] [app] Starting request: urn:request:cca6acbc-4628-4567-a49a-cc09a4cdc4b0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:29,238 [245] [DEBUG] [app] Ending request: urn:request:cca6acbc-4628-4567-a49a-cc09a4cdc4b0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cca6acbc-4628-4567-a49a-cc09a4cdc4b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:23:29,238 [246] [DEBUG] [app] Ending request: urn:request:9a3c7010-50ed-4dde-9f34-64f41160aa13 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a3c7010-50ed-4dde-9f34-64f41160aa13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:23:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:23:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:29,241 [236] [DEBUG] [app] Starting request: urn:request:409d6104-d72f-4b3a-951e-7104d1632af6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:29,241 [236] [DEBUG] [app] Ending request: urn:request:409d6104-d72f-4b3a-951e-7104d1632af6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:409d6104-d72f-4b3a-951e-7104d1632af6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:23:29,241 [238] [DEBUG] [app] Starting request: urn:request:009d73fc-9684-41b6-9629-f7a8826f0c5a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:29,242 [238] [DEBUG] [app] Ending request: urn:request:009d73fc-9684-41b6-9629-f7a8826f0c5a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:009d73fc-9684-41b6-9629-f7a8826f0c5a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:23:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:23:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:23:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:23:29,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:23:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:29,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:29,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:29,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:29,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:29,251 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:29,253 [237] [DEBUG] [app] Ending request: urn:request:77d0fb98-280a-42c2-8550-c19d27f0c169 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77d0fb98-280a-42c2-8550-c19d27f0c169', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:29,253 [236] [DEBUG] [app] Ending request: urn:request:c49ab8a3-fa48-4688-999a-87a1bbc4f0b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c49ab8a3-fa48-4688-999a-87a1bbc4f0b9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:29,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:23:29,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
namespacegcworker stdout | 2025-02-07 05:23:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:23:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:32.625945+00:00 (in 1.004323 seconds)
namespacegcworker stdout | 2025-02-07 05:23:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:31 UTC)" (scheduled at 2025-02-07 05:23:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:23:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:23:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 31, 621929), True, datetime.datetime(2025, 2, 7, 5, 23, 31, 621929), 0, 'namespacegc/%', 50, 1, 0])
storagereplication stdout | 2025-02-07 05:23:31,627 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:23:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:23:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:23:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:23:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:23:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:03.419110+00:00 (in 30.997861 seconds)
repositorygcworker stdout | 2025-02-07 05:23:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:28:32 UTC)" (scheduled at 2025-02-07 05:23:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:23:32,422 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 23, 32, 421549), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:23:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 23, 32, 421549), True, datetime.datetime(2025, 2, 7, 5, 23, 32, 421549), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:23:32,431 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 23, 32, 421549), True, datetime.datetime(2025, 2, 7, 5, 23, 32, 421549), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 23, 32, 421549), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:23:32,432 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:23:32,432 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:28:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:23:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:23:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:31.621131+00:00 (in 58.994640 seconds)
namespacegcworker stdout | 2025-02-07 05:23:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:32 UTC)" (scheduled at 2025-02-07 05:23:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:23:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:23:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:23:33,394 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:23:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:23:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:03.919286+00:00 (in 29.999499 seconds)
buildlogsarchiver stdout | 2025-02-07 05:23:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:03 UTC)" (scheduled at 2025-02-07 05:23:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:23:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 23, 33, 920062), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:23:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:23:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:23:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:23:34,488 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:56.713547+00:00 (in 18.997191 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:37 UTC)" (scheduled at 2025-02-07 05:23:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:23:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:23:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:48.732147+00:00 (in 9.999552 seconds)
notificationworker stdout | 2025-02-07 05:23:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:48 UTC)" (scheduled at 2025-02-07 05:23:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:23:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:23:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 38, 732908), True, datetime.datetime(2025, 2, 7, 5, 23, 38, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:23:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:23:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:23:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:23:38,808 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:23:39,784 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:23:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:23:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:54.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 05:23:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:10 UTC)" (scheduled at 2025-02-07 05:23:40.424250+00:00)
gcworker stdout | 2025-02-07 05:23:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:23:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905520431, None, 1, 0])
gcworker stdout | 2025-02-07 05:23:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:23:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:23:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:23:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:11.831521+00:00 (in 29.999538 seconds)
autopruneworker stdout | 2025-02-07 05:23:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:11 UTC)" (scheduled at 2025-02-07 05:23:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:23:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902221836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:23:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:23:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:23:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:23:43,718 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:23:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:23:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:59.225378+00:00 (in 14.999569 seconds)
quotatotalworker stdout | 2025-02-07 05:23:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:59 UTC)" (scheduled at 2025-02-07 05:23:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:23:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:23:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:23:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:23:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:23:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:23:44,236 [235] [DEBUG] [app] Starting request: urn:request:2b93e950-f317-498e-89d3-33b43c824e9c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:44,236 [237] [DEBUG] [app] Starting request: urn:request:2d07f35c-af11-4708-8198-41646ff4d204 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:44,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:23:44,239 [246] [DEBUG] [app] Starting request: urn:request:81fbf234-6c82-4075-90d1-b13f37a907dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:44,240 [239] [DEBUG] [app] Starting request: urn:request:1372dfca-dd93-4662-a91c-f46feddb1c27 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:44,240 [246] [DEBUG] [app] Ending request: urn:request:81fbf234-6c82-4075-90d1-b13f37a907dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81fbf234-6c82-4075-90d1-b13f37a907dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:23:44,240 [239] [DEBUG] [app] Ending request: urn:request:1372dfca-dd93-4662-a91c-f46feddb1c27 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1372dfca-dd93-4662-a91c-f46feddb1c27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:23:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:23:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:23:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:44,243 [238] [DEBUG] [app] Starting request: urn:request:7d586286-ada4-4952-8f07-df33146d23eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:44,243 [236] [DEBUG] [app] Starting request: urn:request:08a7e18f-529b-4c3a-b128-a1c427cc54c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:44,243 [238] [DEBUG] [app] Ending request: urn:request:7d586286-ada4-4952-8f07-df33146d23eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d586286-ada4-4952-8f07-df33146d23eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:23:44,243 [236] [DEBUG] [app] Ending request: urn:request:08a7e18f-529b-4c3a-b128-a1c427cc54c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:08a7e18f-529b-4c3a-b128-a1c427cc54c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:23:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:23:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:23:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:23:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:23:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:23:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:44,255 [235] [DEBUG] [app] Ending request: urn:request:2b93e950-f317-498e-89d3-33b43c824e9c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b93e950-f317-498e-89d3-33b43c824e9c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:44,255 [237] [DEBUG] [app] Ending request: urn:request:2d07f35c-af11-4708-8198-41646ff4d204 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d07f35c-af11-4708-8198-41646ff4d204', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:23:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
builder stdout | 2025-02-07 05:23:44,295 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:23:44,947 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:23:46,331 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:23:47,079 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:23:47,158 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:23:47,659 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:23:47,662 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:23:47,666 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:23:47,669 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:23:47,671 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:23:48,387 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:23:48,683 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:23:48,686 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:23:48,689 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:48,715 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
notificationworker stdout | 2025-02-07 05:23:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:23:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:50.814677+00:00 (in 2.082145 seconds)
notificationworker stdout | 2025-02-07 05:23:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:58 UTC)" (scheduled at 2025-02-07 05:23:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:23:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:23:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 48, 732778), True, datetime.datetime(2025, 2, 7, 5, 23, 48, 732778), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:23:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:23:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:23:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:23:58 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:23:49,183 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:23:49,253 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:23:49,473 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:23:49,822 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:23:50,079 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:23:50,081 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:23:50,085 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:23:50,088 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:23:50,091 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:23:50,094 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:23:50,096 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:23:50,099 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:23:50,104 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:23:50,211 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:23:50,477 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:23:50,669 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:23:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:23:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:23:58.732147+00:00 (in 7.917018 seconds)
notificationworker stdout | 2025-02-07 05:23:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:50 UTC)" (scheduled at 2025-02-07 05:23:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:23:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:23:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:23:52,130 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:23:52,258 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:23:52,650 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:23:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:23:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:10.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 05:23:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:24 UTC)" (scheduled at 2025-02-07 05:23:54.423055+00:00)
gcworker stdout | 2025-02-07 05:23:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:23:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:23:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:23:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:56.118855+00:00 (in 59.999487 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:23:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:56 UTC)" (scheduled at 2025-02-07 05:23:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:23:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:23:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:23:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:37.715878+00:00 (in 41.001874 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:56 UTC)" (scheduled at 2025-02-07 05:23:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:23:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:23:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:23:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:08.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 05:23:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:08 UTC)" (scheduled at 2025-02-07 05:23:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:23:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:23:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 23, 58, 732908), True, datetime.datetime(2025, 2, 7, 5, 23, 58, 732908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:23:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:23:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:23:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:23:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:23:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:14.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 05:23:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:14 UTC)" (scheduled at 2025-02-07 05:23:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:23:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:23:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:23:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:23:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:23:59,236 [235] [DEBUG] [app] Starting request: urn:request:57870de9-43e1-457d-9872-1457498006f4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:59,236 [238] [DEBUG] [app] Starting request: urn:request:2b020aa2-d9fe-4312-a181-66762dc0ab63 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:23:59,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:23:59,240 [239] [DEBUG] [app] Starting request: urn:request:2c5379d5-489d-4cba-ae31-96bdb3d3b52b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:59,240 [245] [DEBUG] [app] Starting request: urn:request:c6c6904b-2162-44aa-ae61-bc0ddccd785b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:23:59,240 [239] [DEBUG] [app] Ending request: urn:request:2c5379d5-489d-4cba-ae31-96bdb3d3b52b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c5379d5-489d-4cba-ae31-96bdb3d3b52b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:23:59,240 [245] [DEBUG] [app] Ending request: urn:request:c6c6904b-2162-44aa-ae61-bc0ddccd785b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c6c6904b-2162-44aa-ae61-bc0ddccd785b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:23:59,241 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:23:59,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:59,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:59,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:59,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:59,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:23:59,244 [237] [DEBUG] [app] Starting request: urn:request:6585e605-113a-4712-951b-51bd981b7846 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:59,244 [236] [DEBUG] [app] Starting request: urn:request:d6e4bd09-32d8-49f5-90b2-73d357aa28a2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:23:59,245 [237] [DEBUG] [app] Ending request: urn:request:6585e605-113a-4712-951b-51bd981b7846 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6585e605-113a-4712-951b-51bd981b7846', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:23:59,245 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:23:59,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:59,245 [236] [DEBUG] [app] Ending request: urn:request:d6e4bd09-32d8-49f5-90b2-73d357aa28a2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d6e4bd09-32d8-49f5-90b2-73d357aa28a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:23:59,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:23:59,246 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:23:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:23:59,246 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:23:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:23:59,246 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:23:59,247 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:23:59,247 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:23:59,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:59,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:59,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:23:59,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:23:59,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:59,255 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:23:59,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:59,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:23:59,257 [238] [DEBUG] [app] Ending request: urn:request:2b020aa2-d9fe-4312-a181-66762dc0ab63 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b020aa2-d9fe-4312-a181-66762dc0ab63', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:59,257 [235] [DEBUG] [app] Ending request: urn:request:57870de9-43e1-457d-9872-1457498006f4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:57870de9-43e1-457d-9872-1457498006f4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:23:59,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:59,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:23:59,258 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 05:23:59,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:23:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:23:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
exportactionlogsworker stdout | 2025-02-07 05:23:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:23:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:04.936688+00:00 (in 4.998333 seconds)
exportactionlogsworker stdout | 2025-02-07 05:23:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:59 UTC)" (scheduled at 2025-02-07 05:23:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:23:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:23:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:24:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:24:01,640 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositoryactioncounter stdout | 2025-02-07 05:24:03,401 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
repositorygcworker stdout | 2025-02-07 05:24:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:24:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:23.420886+00:00 (in 20.001300 seconds)
repositorygcworker stdout | 2025-02-07 05:24:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:03 UTC)" (scheduled at 2025-02-07 05:24:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:24:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:24:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 3, 419855), True, datetime.datetime(2025, 2, 7, 5, 24, 3, 419855), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:24:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:24:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:24:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:24:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:24:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:33.919286+00:00 (in 29.999530 seconds)
buildlogsarchiver stdout | 2025-02-07 05:24:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:33 UTC)" (scheduled at 2025-02-07 05:24:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:24:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 24, 3, 920005), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:24:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:24:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:24:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:24:04,524 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:24:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:24:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:52.937648+00:00 (in 48.000508 seconds)
exportactionlogsworker stdout | 2025-02-07 05:24:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:04 UTC)" (scheduled at 2025-02-07 05:24:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:24:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:24:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 4, 937441), True, datetime.datetime(2025, 2, 7, 5, 24, 4, 937441), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:24:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:24:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:24:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:24:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:24:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:18.732147+00:00 (in 9.999630 seconds)
notificationworker stdout | 2025-02-07 05:24:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:18 UTC)" (scheduled at 2025-02-07 05:24:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:24:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:24:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 8, 732785), True, datetime.datetime(2025, 2, 7, 5, 24, 8, 732785), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:24:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:24:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:24:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:24:08,837 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:24:09,823 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:24:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:24:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:24.423055+00:00 (in 13.998348 seconds)
gcworker stdout | 2025-02-07 05:24:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:40 UTC)" (scheduled at 2025-02-07 05:24:10.424250+00:00)
gcworker stdout | 2025-02-07 05:24:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:24:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905550431, None, 1, 0])
gcworker stdout | 2025-02-07 05:24:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:24:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:24:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:24:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:41.831521+00:00 (in 29.999572 seconds)
autopruneworker stdout | 2025-02-07 05:24:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:41 UTC)" (scheduled at 2025-02-07 05:24:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:24:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902251836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:24:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:24:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:24:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:24:13,746 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:24:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:24:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:29.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 05:24:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:29 UTC)" (scheduled at 2025-02-07 05:24:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:24:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:24:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:24:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:24:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:24:14,235 [237] [DEBUG] [app] Starting request: urn:request:e0cf55f2-7576-446c-896b-ab98e51d971b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:14,235 [236] [DEBUG] [app] Starting request: urn:request:dd331e88-fa7b-4e0c-bb30-51b33bd2fe4e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:24:14,239 [246] [DEBUG] [app] Starting request: urn:request:7c419bf8-8172-414d-bfd9-c6eb2025795f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:14,239 [244] [DEBUG] [app] Starting request: urn:request:b3fde5ef-e46c-47d5-96cd-898189f280af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:14,239 [244] [DEBUG] [app] Ending request: urn:request:b3fde5ef-e46c-47d5-96cd-898189f280af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b3fde5ef-e46c-47d5-96cd-898189f280af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:24:14,239 [246] [DEBUG] [app] Ending request: urn:request:7c419bf8-8172-414d-bfd9-c6eb2025795f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7c419bf8-8172-414d-bfd9-c6eb2025795f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:24:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:24:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:24:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:14,242 [236] [DEBUG] [app] Starting request: urn:request:cb9d7906-144c-4c44-8d42-2845986dd3d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:14,242 [237] [DEBUG] [app] Starting request: urn:request:f9503e72-d693-44be-9e90-721bd68ea727 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:14,242 [236] [DEBUG] [app] Ending request: urn:request:cb9d7906-144c-4c44-8d42-2845986dd3d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cb9d7906-144c-4c44-8d42-2845986dd3d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:14,242 [237] [DEBUG] [app] Ending request: urn:request:f9503e72-d693-44be-9e90-721bd68ea727 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9503e72-d693-44be-9e90-721bd68ea727', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:24:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:24:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:14,254 [237] [DEBUG] [app] Ending request: urn:request:e0cf55f2-7576-446c-896b-ab98e51d971b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0cf55f2-7576-446c-896b-ab98e51d971b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:14,254 [236] [DEBUG] [app] Ending request: urn:request:dd331e88-fa7b-4e0c-bb30-51b33bd2fe4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dd331e88-fa7b-4e0c-bb30-51b33bd2fe4e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:24:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:24:14,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:24:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
builder stdout | 2025-02-07 05:24:14,332 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:24:14,966 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:24:16,351 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:24:17,089 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:24:17,166 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:24:17,667 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:24:17,670 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:24:17,674 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:24:17,676 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:24:17,679 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:24:18,423 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:24:18,690 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:24:18,694 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:24:18,696 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:24:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:24:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:28.732147+00:00 (in 9.999583 seconds)
notificationworker stdout | 2025-02-07 05:24:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:28 UTC)" (scheduled at 2025-02-07 05:24:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:24:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:24:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 18, 732878), True, datetime.datetime(2025, 2, 7, 5, 24, 18, 732878), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:24:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:24:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:24:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:28 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:18,750 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:24:19,219 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:24:19,289 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:24:19,510 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:24:19,859 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:24:20,088 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:24:20,091 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:24:20,094 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:24:20,096 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:24:20,100 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:24:20,104 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:24:20,107 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:24:20,111 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:24:20,113 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:24:20,234 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:24:20,513 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:24:20,676 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:24:22,166 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:24:22,266 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:24:22,682 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:24:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:24:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:03.419110+00:00 (in 39.997760 seconds)
repositorygcworker stdout | 2025-02-07 05:24:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:23 UTC)" (scheduled at 2025-02-07 05:24:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:24:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:24:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:24:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:24:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:40.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 05:24:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:54 UTC)" (scheduled at 2025-02-07 05:24:24.423055+00:00)
gcworker stdout | 2025-02-07 05:24:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:24:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:24:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:24:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:24:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:38.732147+00:00 (in 9.999584 seconds)
notificationworker stdout | 2025-02-07 05:24:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:38 UTC)" (scheduled at 2025-02-07 05:24:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:24:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:24:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 28, 732830), True, datetime.datetime(2025, 2, 7, 5, 24, 28, 732830), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:24:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:24:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:24:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:24:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:24:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:44.225378+00:00 (in 14.999578 seconds)
quotatotalworker stdout | 2025-02-07 05:24:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:44 UTC)" (scheduled at 2025-02-07 05:24:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:24:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:24:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:24:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:24:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:24:29,235 [236] [DEBUG] [app] Starting request: urn:request:80bd7645-998a-4fe0-86ad-18b771062164 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:29,235 [235] [DEBUG] [app] Starting request: urn:request:49a25916-5b1e-4019-9d10-bf14a6bc60d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:24:29,238 [239] [DEBUG] [app] Starting request: urn:request:770daebe-1696-4e5f-8ad3-2c5ed54ab41f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:29,238 [245] [DEBUG] [app] Starting request: urn:request:f40f25e8-296a-4f4c-b76e-ab43f83b5065 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:29,238 [239] [DEBUG] [app] Ending request: urn:request:770daebe-1696-4e5f-8ad3-2c5ed54ab41f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:770daebe-1696-4e5f-8ad3-2c5ed54ab41f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:24:29,238 [245] [DEBUG] [app] Ending request: urn:request:f40f25e8-296a-4f4c-b76e-ab43f83b5065 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f40f25e8-296a-4f4c-b76e-ab43f83b5065', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:24:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:24:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:24:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:29,241 [238] [DEBUG] [app] Starting request: urn:request:277b7e28-6c59-4b1d-a0bf-a6230e5e3633 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:29,241 [237] [DEBUG] [app] Starting request: urn:request:0d7e6d8e-32e4-4973-9694-53c2eaa7e1d2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:29,242 [238] [DEBUG] [app] Ending request: urn:request:277b7e28-6c59-4b1d-a0bf-a6230e5e3633 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:277b7e28-6c59-4b1d-a0bf-a6230e5e3633', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:29,242 [237] [DEBUG] [app] Ending request: urn:request:0d7e6d8e-32e4-4973-9694-53c2eaa7e1d2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d7e6d8e-32e4-4973-9694-53c2eaa7e1d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:24:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:24:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:29,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:29,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:29,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:29,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:29,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:29,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:29,253 [235] [DEBUG] [app] Ending request: urn:request:49a25916-5b1e-4019-9d10-bf14a6bc60d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:49a25916-5b1e-4019-9d10-bf14a6bc60d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:29,253 [236] [DEBUG] [app] Ending request: urn:request:80bd7645-998a-4fe0-86ad-18b771062164 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:80bd7645-998a-4fe0-86ad-18b771062164', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:24:29,253 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:24:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:24:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:24:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:24:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:32.625945+00:00 (in 1.004339 seconds)
namespacegcworker stdout | 2025-02-07 05:24:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:31 UTC)" (scheduled at 2025-02-07 05:24:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:24:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:24:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 31, 621878), True, datetime.datetime(2025, 2, 7, 5, 24, 31, 621878), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:24:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:24:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:24:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:24:31,676 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:24:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:24:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:31.621131+00:00 (in 58.994734 seconds)
namespacegcworker stdout | 2025-02-07 05:24:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:32 UTC)" (scheduled at 2025-02-07 05:24:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:24:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:24:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:24:33,437 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:24:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:24:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:03.919286+00:00 (in 29.999532 seconds)
buildlogsarchiver stdout | 2025-02-07 05:24:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:03 UTC)" (scheduled at 2025-02-07 05:24:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:24:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 24, 33, 919940), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:24:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:24:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:24:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:24:34,560 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:56.713547+00:00 (in 18.997183 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:37 UTC)" (scheduled at 2025-02-07 05:24:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:24:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:24:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:48.732147+00:00 (in 9.999586 seconds)
notificationworker stdout | 2025-02-07 05:24:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:48 UTC)" (scheduled at 2025-02-07 05:24:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:24:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:24:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 38, 732817), True, datetime.datetime(2025, 2, 7, 5, 24, 38, 732817), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:24:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:24:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:24:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:24:38,873 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:24:39,843 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:24:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:24:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:54.423055+00:00 (in 13.998359 seconds)
gcworker stdout | 2025-02-07 05:24:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:10 UTC)" (scheduled at 2025-02-07 05:24:40.424250+00:00)
gcworker stdout | 2025-02-07 05:24:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:24:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905580431, None, 1, 0])
gcworker stdout | 2025-02-07 05:24:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:24:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:24:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:24:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:11.831521+00:00 (in 29.999562 seconds)
autopruneworker stdout | 2025-02-07 05:24:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:11 UTC)" (scheduled at 2025-02-07 05:24:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:24:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902281836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:24:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:24:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:24:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:24:43,782 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:24:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:24:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:59.225378+00:00 (in 14.999595 seconds)
quotatotalworker stdout | 2025-02-07 05:24:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:59 UTC)" (scheduled at 2025-02-07 05:24:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:24:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:24:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:24:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:24:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:24:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:24:44,235 [238] [DEBUG] [app] Starting request: urn:request:128f60ec-8077-4e44-8600-5e2ced9b71d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:44,235 [237] [DEBUG] [app] Starting request: urn:request:629af619-cea4-47a9-b7d1-d9ef8a2b1e14 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:24:44,238 [246] [DEBUG] [app] Starting request: urn:request:810334cc-1e13-4737-8ca1-4b4a6d852c20 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:44,238 [239] [DEBUG] [app] Starting request: urn:request:3692a735-3a68-4d66-9c6b-23d3014a345e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:44,238 [246] [DEBUG] [app] Ending request: urn:request:810334cc-1e13-4737-8ca1-4b4a6d852c20 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:810334cc-1e13-4737-8ca1-4b4a6d852c20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:24:44,238 [239] [DEBUG] [app] Ending request: urn:request:3692a735-3a68-4d66-9c6b-23d3014a345e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3692a735-3a68-4d66-9c6b-23d3014a345e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:24:44,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:24:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:24:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:24:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:44,241 [238] [DEBUG] [app] Starting request: urn:request:0ed244f6-2242-4ffc-b248-d8ae5d47fa31 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:44,242 [238] [DEBUG] [app] Ending request: urn:request:0ed244f6-2242-4ffc-b248-d8ae5d47fa31 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0ed244f6-2242-4ffc-b248-d8ae5d47fa31', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:24:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:24:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:44,243 [235] [DEBUG] [app] Starting request: urn:request:fd6ff290-cb9c-4ba3-b398-c03886facba1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:44,244 [235] [DEBUG] [app] Ending request: urn:request:fd6ff290-cb9c-4ba3-b398-c03886facba1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd6ff290-cb9c-4ba3-b398-c03886facba1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:44,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:24:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:44,253 [237] [DEBUG] [app] Ending request: urn:request:629af619-cea4-47a9-b7d1-d9ef8a2b1e14 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:629af619-cea4-47a9-b7d1-d9ef8a2b1e14', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:24:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:24:44,255 [238] [DEBUG] [app] Ending request: urn:request:128f60ec-8077-4e44-8600-5e2ced9b71d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:128f60ec-8077-4e44-8600-5e2ced9b71d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:24:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
builder stdout | 2025-02-07 05:24:44,354 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:24:45,002 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:24:46,369 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:24:47,109 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:24:47,203 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:24:47,675 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:24:47,679 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:24:47,683 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:24:47,686 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:24:47,689 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:24:48,445 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:24:48,698 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:24:48,702 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:24:48,704 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:24:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:24:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:50.814677+00:00 (in 2.082124 seconds)
notificationworker stdout | 2025-02-07 05:24:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:58 UTC)" (scheduled at 2025-02-07 05:24:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:24:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:24:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 48, 732869), True, datetime.datetime(2025, 2, 7, 5, 24, 48, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:24:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:24:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:24:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:24:58 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:48,786 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:24:49,234 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:24:49,317 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:24:49,521 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:24:49,895 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:24:50,098 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:24:50,100 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:24:50,103 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:24:50,106 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:24:50,111 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:24:50,116 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:24:50,118 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:24:50,121 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:24:50,123 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:24:50,244 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:24:50,549 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:24:50,712 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:24:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:24:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:58.732147+00:00 (in 7.916949 seconds)
notificationworker stdout | 2025-02-07 05:24:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:50 UTC)" (scheduled at 2025-02-07 05:24:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:24:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:24:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:24:52,195 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:24:52,302 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:24:52,718 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
exportactionlogsworker stdout | 2025-02-07 05:24:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:24:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:24:59.937899+00:00 (in 6.999791 seconds)
exportactionlogsworker stdout | 2025-02-07 05:24:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:29:52 UTC)" (scheduled at 2025-02-07 05:24:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:24:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 24, 52, 938411), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:24:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 24, 52, 938411), True, datetime.datetime(2025, 2, 7, 5, 24, 52, 938411), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:24:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 24, 52, 938411), True, datetime.datetime(2025, 2, 7, 5, 24, 52, 938411), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 24, 52, 938411), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:24:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:24:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:29:52 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:24:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:24:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:10.424250+00:00 (in 16.000705 seconds)
gcworker stdout | 2025-02-07 05:24:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:24 UTC)" (scheduled at 2025-02-07 05:24:54.423055+00:00)
gcworker stdout | 2025-02-07 05:24:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:24:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:24:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:24:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:56.118855+00:00 (in 59.999497 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:24:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:56 UTC)" (scheduled at 2025-02-07 05:24:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:24:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:24:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:24:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:37.715878+00:00 (in 41.001870 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:56 UTC)" (scheduled at 2025-02-07 05:24:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:24:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:24:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:24:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:08.732147+00:00 (in 9.999510 seconds)
notificationworker stdout | 2025-02-07 05:24:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:08 UTC)" (scheduled at 2025-02-07 05:24:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:24:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:24:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 24, 58, 732936), True, datetime.datetime(2025, 2, 7, 5, 24, 58, 732936), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:24:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:24:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:24:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:24:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:24:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:14.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 05:24:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:14 UTC)" (scheduled at 2025-02-07 05:24:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:24:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:24:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:24:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:24:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:24:59,236 [238] [DEBUG] [app] Starting request: urn:request:638e8fbc-2df7-47e6-9e29-8d6e20ec6cbc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:59,236 [235] [DEBUG] [app] Starting request: urn:request:4801d612-a4b8-41de-b1b5-a75dd5b052f0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:24:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:24:59,239 [239] [DEBUG] [app] Starting request: urn:request:eccfd220-9b99-4be7-83c5-699b7efca43d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:59,239 [246] [DEBUG] [app] Starting request: urn:request:3d603d38-ab95-41e5-b3d1-224c42f5dcce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:24:59,239 [246] [DEBUG] [app] Ending request: urn:request:3d603d38-ab95-41e5-b3d1-224c42f5dcce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3d603d38-ab95-41e5-b3d1-224c42f5dcce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:24:59,239 [239] [DEBUG] [app] Ending request: urn:request:eccfd220-9b99-4be7-83c5-699b7efca43d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eccfd220-9b99-4be7-83c5-699b7efca43d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:24:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:24:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:24:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:24:59,242 [238] [DEBUG] [app] Starting request: urn:request:b2f2be96-09d2-4df8-a2ce-aeae2418f49f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:59,242 [238] [DEBUG] [app] Ending request: urn:request:b2f2be96-09d2-4df8-a2ce-aeae2418f49f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2f2be96-09d2-4df8-a2ce-aeae2418f49f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:24:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:59,243 [236] [DEBUG] [app] Starting request: urn:request:0afbd1e8-1740-47ab-9e16-19d181bb4642 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:24:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:59,243 [236] [DEBUG] [app] Ending request: urn:request:0afbd1e8-1740-47ab-9e16-19d181bb4642 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0afbd1e8-1740-47ab-9e16-19d181bb4642', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:24:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:24:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:24:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:24:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:24:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:24:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:24:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:24:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:24:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:24:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:24:59,254 [238] [DEBUG] [app] Ending request: urn:request:638e8fbc-2df7-47e6-9e29-8d6e20ec6cbc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:638e8fbc-2df7-47e6-9e29-8d6e20ec6cbc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:24:59,254 [235] [DEBUG] [app] Ending request: urn:request:4801d612-a4b8-41de-b1b5-a75dd5b052f0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4801d612-a4b8-41de-b1b5-a75dd5b052f0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:24:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:24:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:24:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:24:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:24:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:24:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:24:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:04.936688+00:00 (in 4.998235 seconds)
exportactionlogsworker stdout | 2025-02-07 05:24:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:59 UTC)" (scheduled at 2025-02-07 05:24:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:24:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:24:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:25:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:25:01,711 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:25:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:25:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:23.420886+00:00 (in 20.001329 seconds)
repositorygcworker stdout | 2025-02-07 05:25:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:03 UTC)" (scheduled at 2025-02-07 05:25:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:25:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:25:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 3, 419833), True, datetime.datetime(2025, 2, 7, 5, 25, 3, 419833), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:25:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:25:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:25:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:25:03,444 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:25:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:25:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:33.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 05:25:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:33 UTC)" (scheduled at 2025-02-07 05:25:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:25:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 25, 3, 919981), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:25:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:25:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:25:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:25:04,573 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:25:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:25:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:59.937899+00:00 (in 55.000759 seconds)
exportactionlogsworker stdout | 2025-02-07 05:25:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:04 UTC)" (scheduled at 2025-02-07 05:25:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:25:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:25:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 4, 937493), True, datetime.datetime(2025, 2, 7, 5, 25, 4, 937493), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:25:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:25:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:25:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:25:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:25:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:18.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 05:25:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:18 UTC)" (scheduled at 2025-02-07 05:25:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:25:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:25:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 8, 732886), True, datetime.datetime(2025, 2, 7, 5, 25, 8, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:25:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:25:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:25:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:25:08,907 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:25:09,859 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:25:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:25:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:24.423055+00:00 (in 13.998328 seconds)
gcworker stdout | 2025-02-07 05:25:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:40 UTC)" (scheduled at 2025-02-07 05:25:10.424250+00:00)
gcworker stdout | 2025-02-07 05:25:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:25:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905610431, None, 1, 0])
gcworker stdout | 2025-02-07 05:25:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:25:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:25:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:25:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:41.831521+00:00 (in 29.999573 seconds)
autopruneworker stdout | 2025-02-07 05:25:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:41 UTC)" (scheduled at 2025-02-07 05:25:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:25:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902311836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:25:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:25:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:25:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:25:13,818 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:25:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:25:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:29.225378+00:00 (in 14.999558 seconds)
quotatotalworker stdout | 2025-02-07 05:25:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:29 UTC)" (scheduled at 2025-02-07 05:25:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:25:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:25:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:25:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:25:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:25:14,235 [235] [DEBUG] [app] Starting request: urn:request:139a967d-306c-4a79-bb70-2babcf7d723a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:14,235 [237] [DEBUG] [app] Starting request: urn:request:e3ad4ad4-7859-4f66-b58f-d0d8eea43ea9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:25:14,238 [246] [DEBUG] [app] Starting request: urn:request:f50cae75-9a3a-47f9-8685-011554909228 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:14,238 [239] [DEBUG] [app] Starting request: urn:request:0c0dff4d-0034-4679-810a-9c3a74cc69aa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:14,238 [239] [DEBUG] [app] Ending request: urn:request:0c0dff4d-0034-4679-810a-9c3a74cc69aa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c0dff4d-0034-4679-810a-9c3a74cc69aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:25:14,238 [246] [DEBUG] [app] Ending request: urn:request:f50cae75-9a3a-47f9-8685-011554909228 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f50cae75-9a3a-47f9-8685-011554909228', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:25:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:25:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:14,241 [235] [DEBUG] [app] Starting request: urn:request:0f86b845-3dff-44ce-8852-ac1bde8cfdc5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:14,241 [235] [DEBUG] [app] Ending request: urn:request:0f86b845-3dff-44ce-8852-ac1bde8cfdc5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0f86b845-3dff-44ce-8852-ac1bde8cfdc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:25:14,241 [238] [DEBUG] [app] Starting request: urn:request:2e305c52-bbe8-46c1-b2ff-23c56770c709 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:14,242 [238] [DEBUG] [app] Ending request: urn:request:2e305c52-bbe8-46c1-b2ff-23c56770c709 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e305c52-bbe8-46c1-b2ff-23c56770c709', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:25:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:25:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:14,253 [235] [DEBUG] [app] Ending request: urn:request:139a967d-306c-4a79-bb70-2babcf7d723a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:139a967d-306c-4a79-bb70-2babcf7d723a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:25:14,253 [237] [DEBUG] [app] Ending request: urn:request:e3ad4ad4-7859-4f66-b58f-d0d8eea43ea9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3ad4ad4-7859-4f66-b58f-d0d8eea43ea9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:25:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:25:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:25:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:25:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
builder stdout | 2025-02-07 05:25:14,389 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:25:15,030 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:25:16,379 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:25:17,140 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:25:17,240 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:25:17,683 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:25:17,688 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:25:17,692 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:25:17,695 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:25:17,698 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:25:18,466 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:25:18,706 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:25:18,710 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:25:18,713 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:25:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:25:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:28.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 05:25:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:28 UTC)" (scheduled at 2025-02-07 05:25:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:25:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:25:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 18, 732843), True, datetime.datetime(2025, 2, 7, 5, 25, 18, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:25:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:25:18,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:25:18,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:28 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:18,818 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:25:19,264 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:25:19,328 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:25:19,540 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:25:19,928 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:25:20,107 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:25:20,111 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:25:20,113 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:25:20,116 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:25:20,120 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:25:20,128 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:25:20,130 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:25:20,133 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:25:20,135 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gcworker stdout | 2025-02-07 05:25:20,263 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:25:20,578 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:25:20,749 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:25:22,231 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:25:22,315 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:25:22,754 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:25:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:25:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:03.419110+00:00 (in 39.997726 seconds)
repositorygcworker stdout | 2025-02-07 05:25:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:23 UTC)" (scheduled at 2025-02-07 05:25:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:25:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:25:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:25:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:25:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:40.424250+00:00 (in 16.000713 seconds)
gcworker stdout | 2025-02-07 05:25:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:54 UTC)" (scheduled at 2025-02-07 05:25:24.423055+00:00)
gcworker stdout | 2025-02-07 05:25:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:25:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:25:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:25:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:25:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:38.732147+00:00 (in 9.999545 seconds)
notificationworker stdout | 2025-02-07 05:25:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:38 UTC)" (scheduled at 2025-02-07 05:25:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:25:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:25:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 28, 732898), True, datetime.datetime(2025, 2, 7, 5, 25, 28, 732898), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:25:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:25:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:25:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:25:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:25:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:44.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 05:25:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:44 UTC)" (scheduled at 2025-02-07 05:25:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:25:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:25:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:25:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:25:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:25:29,236 [235] [DEBUG] [app] Starting request: urn:request:06c783ff-aa1c-4870-8aaf-c0862608032d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:29,236 [238] [DEBUG] [app] Starting request: urn:request:10c7ea29-a56b-4ae7-af40-c4648353a1a1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:25:29,239 [239] [DEBUG] [app] Starting request: urn:request:9b6f0841-93d6-40d5-bb89-525754a5e8eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:29,239 [246] [DEBUG] [app] Starting request: urn:request:8a7edeb0-fcf0-4ca1-9e4c-199328d27a23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:29,240 [246] [DEBUG] [app] Ending request: urn:request:8a7edeb0-fcf0-4ca1-9e4c-199328d27a23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a7edeb0-fcf0-4ca1-9e4c-199328d27a23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:25:29,240 [239] [DEBUG] [app] Ending request: urn:request:9b6f0841-93d6-40d5-bb89-525754a5e8eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b6f0841-93d6-40d5-bb89-525754a5e8eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:25:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:25:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:29,243 [237] [DEBUG] [app] Starting request: urn:request:f4c6ae26-5b67-4b08-9dd2-0f40e43ebe51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:29,243 [236] [DEBUG] [app] Starting request: urn:request:1d3c7b29-3986-4495-8ae3-356c365b89b8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:29,244 [237] [DEBUG] [app] Ending request: urn:request:f4c6ae26-5b67-4b08-9dd2-0f40e43ebe51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4c6ae26-5b67-4b08-9dd2-0f40e43ebe51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:25:29,244 [236] [DEBUG] [app] Ending request: urn:request:1d3c7b29-3986-4495-8ae3-356c365b89b8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1d3c7b29-3986-4495-8ae3-356c365b89b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:25:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:25:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:29,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:29,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:29,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:29,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:29,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:29,256 [238] [DEBUG] [app] Ending request: urn:request:10c7ea29-a56b-4ae7-af40-c4648353a1a1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10c7ea29-a56b-4ae7-af40-c4648353a1a1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:25:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:25:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:25:29,256 [235] [DEBUG] [app] Ending request: urn:request:06c783ff-aa1c-4870-8aaf-c0862608032d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:06c783ff-aa1c-4870-8aaf-c0862608032d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:25:29,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:25:29,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
namespacegcworker stdout | 2025-02-07 05:25:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:25:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:32.625945+00:00 (in 1.004375 seconds)
namespacegcworker stdout | 2025-02-07 05:25:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:31 UTC)" (scheduled at 2025-02-07 05:25:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:25:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:25:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 31, 621841), True, datetime.datetime(2025, 2, 7, 5, 25, 31, 621841), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:25:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:25:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:25:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:25:31,734 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:25:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:25:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:31.621131+00:00 (in 58.994732 seconds)
namespacegcworker stdout | 2025-02-07 05:25:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:32 UTC)" (scheduled at 2025-02-07 05:25:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:25:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:25:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:25:33,481 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:25:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:25:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:03.919286+00:00 (in 29.999534 seconds)
buildlogsarchiver stdout | 2025-02-07 05:25:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:03 UTC)" (scheduled at 2025-02-07 05:25:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:25:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 25, 33, 920034), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:25:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:25:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:25:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:25:34,592 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:56.713547+00:00 (in 18.997208 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:37 UTC)" (scheduled at 2025-02-07 05:25:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:25:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:25:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:48.732147+00:00 (in 9.999562 seconds)
notificationworker stdout | 2025-02-07 05:25:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:48 UTC)" (scheduled at 2025-02-07 05:25:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:25:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:25:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 38, 732899), True, datetime.datetime(2025, 2, 7, 5, 25, 38, 732899), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:25:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:25:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:25:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:25:38,917 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:25:39,887 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:25:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:25:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:54.423055+00:00 (in 13.998367 seconds)
gcworker stdout | 2025-02-07 05:25:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:10 UTC)" (scheduled at 2025-02-07 05:25:40.424250+00:00)
gcworker stdout | 2025-02-07 05:25:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:25:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905640431, None, 1, 0])
gcworker stdout | 2025-02-07 05:25:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:25:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:25:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:25:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:11.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 05:25:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:11 UTC)" (scheduled at 2025-02-07 05:25:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:25:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902341836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:25:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:25:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:25:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:25:43,854 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:25:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:25:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:59.225378+00:00 (in 14.999576 seconds)
quotatotalworker stdout | 2025-02-07 05:25:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:59 UTC)" (scheduled at 2025-02-07 05:25:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:25:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:25:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:25:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:25:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:25:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:25:44,236 [235] [DEBUG] [app] Starting request: urn:request:a947dc39-0e24-4ee3-bd68-d1d64c9398ba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:44,236 [238] [DEBUG] [app] Starting request: urn:request:9cbf2d0d-2948-4955-aa7f-71e20e04ace0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:25:44,239 [244] [DEBUG] [app] Starting request: urn:request:e57be2eb-5ea3-4967-a80a-6ddace5eb566 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:44,239 [246] [DEBUG] [app] Starting request: urn:request:aaab6078-f2d6-4946-bd95-d452b3b62dbc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:44,239 [244] [DEBUG] [app] Ending request: urn:request:e57be2eb-5ea3-4967-a80a-6ddace5eb566 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e57be2eb-5ea3-4967-a80a-6ddace5eb566', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:25:44,239 [246] [DEBUG] [app] Ending request: urn:request:aaab6078-f2d6-4946-bd95-d452b3b62dbc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aaab6078-f2d6-4946-bd95-d452b3b62dbc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:25:44,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:25:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:25:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:44,242 [238] [DEBUG] [app] Starting request: urn:request:16c2d6dc-c861-4e8d-9c0d-06d69cd2ba37 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:44,242 [238] [DEBUG] [app] Ending request: urn:request:16c2d6dc-c861-4e8d-9c0d-06d69cd2ba37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16c2d6dc-c861-4e8d-9c0d-06d69cd2ba37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:25:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:25:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:44,243 [237] [DEBUG] [app] Starting request: urn:request:87dedc5a-883f-4d2a-88f2-f551a61d08c7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:44,243 [237] [DEBUG] [app] Ending request: urn:request:87dedc5a-883f-4d2a-88f2-f551a61d08c7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87dedc5a-883f-4d2a-88f2-f551a61d08c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:25:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:44,254 [238] [DEBUG] [app] Ending request: urn:request:9cbf2d0d-2948-4955-aa7f-71e20e04ace0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9cbf2d0d-2948-4955-aa7f-71e20e04ace0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:25:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:25:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:25:44,255 [235] [DEBUG] [app] Ending request: urn:request:a947dc39-0e24-4ee3-bd68-d1d64c9398ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a947dc39-0e24-4ee3-bd68-d1d64c9398ba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:25:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 05:25:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
builder stdout | 2025-02-07 05:25:44,425 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:25:45,047 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:25:46,411 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:25:47,167 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:25:47,271 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:25:47,691 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:25:47,695 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:25:47,699 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:25:47,702 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:25:47,704 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:25:48,502 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:25:48,714 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:25:48,718 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:25:48,721 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:25:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:25:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:50.814677+00:00 (in 2.082121 seconds)
notificationworker stdout | 2025-02-07 05:25:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:58 UTC)" (scheduled at 2025-02-07 05:25:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:25:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:25:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 48, 732824), True, datetime.datetime(2025, 2, 7, 5, 25, 48, 732824), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:25:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:25:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:25:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:25:58 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:48,833 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:25:49,293 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:25:49,340 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:25:49,577 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:25:49,965 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:25:50,114 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:25:50,123 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:25:50,126 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:25:50,128 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:25:50,131 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:25:50,136 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:25:50,140 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:25:50,143 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:25:50,145 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gcworker stdout | 2025-02-07 05:25:50,299 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:25:50,596 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:25:50,785 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:25:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:25:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:25:58.732147+00:00 (in 7.917008 seconds)
notificationworker stdout | 2025-02-07 05:25:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:50 UTC)" (scheduled at 2025-02-07 05:25:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:25:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:25:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:25:52,264 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:25:52,331 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:25:52,778 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:25:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:25:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:10.424250+00:00 (in 16.000745 seconds)
gcworker stdout | 2025-02-07 05:25:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:24 UTC)" (scheduled at 2025-02-07 05:25:54.423055+00:00)
gcworker stdout | 2025-02-07 05:25:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:25:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:25:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:25:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:56.118855+00:00 (in 59.999528 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:25:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:56 UTC)" (scheduled at 2025-02-07 05:25:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:25:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:25:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:25:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:37.715878+00:00 (in 41.001884 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:56 UTC)" (scheduled at 2025-02-07 05:25:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:25:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:25:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:25:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:08.732147+00:00 (in 9.999584 seconds)
notificationworker stdout | 2025-02-07 05:25:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:08 UTC)" (scheduled at 2025-02-07 05:25:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:25:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:25:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 25, 58, 732846), True, datetime.datetime(2025, 2, 7, 5, 25, 58, 732846), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:25:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:25:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:25:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:25:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:25:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:14.225378+00:00 (in 14.999573 seconds)
quotatotalworker stdout | 2025-02-07 05:25:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:14 UTC)" (scheduled at 2025-02-07 05:25:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:25:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:25:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:25:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:25:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:25:59,235 [238] [DEBUG] [app] Starting request: urn:request:77ac8cf2-d790-45f7-a1ae-1855ee60089e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:59,235 [235] [DEBUG] [app] Starting request: urn:request:441e74e0-63ff-4b99-b74a-48e98db496fc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:25:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:25:59,238 [244] [DEBUG] [app] Starting request: urn:request:c94355c3-e0e7-4b5f-af59-36b12368c494 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:59,238 [246] [DEBUG] [app] Starting request: urn:request:6f2f2e5b-f8da-49dd-b0a8-6fb7deb42d5c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:25:59,238 [244] [DEBUG] [app] Ending request: urn:request:c94355c3-e0e7-4b5f-af59-36b12368c494 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c94355c3-e0e7-4b5f-af59-36b12368c494', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:25:59,238 [246] [DEBUG] [app] Ending request: urn:request:6f2f2e5b-f8da-49dd-b0a8-6fb7deb42d5c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f2f2e5b-f8da-49dd-b0a8-6fb7deb42d5c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:25:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:25:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:25:59,242 [237] [DEBUG] [app] Starting request: urn:request:b7bc1235-3829-496d-9845-904fb2e7f5d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:59,242 [236] [DEBUG] [app] Starting request: urn:request:e4bbd7a5-abb6-4e83-bff6-2d28b91ae887 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:25:59,242 [236] [DEBUG] [app] Ending request: urn:request:e4bbd7a5-abb6-4e83-bff6-2d28b91ae887 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4bbd7a5-abb6-4e83-bff6-2d28b91ae887', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:25:59,242 [237] [DEBUG] [app] Ending request: urn:request:b7bc1235-3829-496d-9845-904fb2e7f5d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7bc1235-3829-496d-9845-904fb2e7f5d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:25:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:25:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:25:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:25:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:25:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:25:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:25:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:25:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:25:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:25:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:25:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:25:59,254 [235] [DEBUG] [app] Ending request: urn:request:441e74e0-63ff-4b99-b74a-48e98db496fc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:441e74e0-63ff-4b99-b74a-48e98db496fc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:25:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:25:59,254 [238] [DEBUG] [app] Ending request: urn:request:77ac8cf2-d790-45f7-a1ae-1855ee60089e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77ac8cf2-d790-45f7-a1ae-1855ee60089e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:25:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:25:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:25:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:25:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:25:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 05:25:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:25:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:04.936688+00:00 (in 4.998312 seconds)
exportactionlogsworker stdout | 2025-02-07 05:25:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:59 UTC)" (scheduled at 2025-02-07 05:25:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:25:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:25:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:26:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:26:01,754 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:26:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:26:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:23.420886+00:00 (in 20.001324 seconds)
repositorygcworker stdout | 2025-02-07 05:26:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:03 UTC)" (scheduled at 2025-02-07 05:26:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:26:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:26:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 3, 419829), True, datetime.datetime(2025, 2, 7, 5, 26, 3, 419829), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:26:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:26:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:26:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:26:03,505 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:26:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:26:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:33.919286+00:00 (in 29.999554 seconds)
buildlogsarchiver stdout | 2025-02-07 05:26:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:33 UTC)" (scheduled at 2025-02-07 05:26:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:26:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 26, 3, 920006), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:26:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:26:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:26:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:26:04,610 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:26:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:26:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:59.937899+00:00 (in 55.000770 seconds)
exportactionlogsworker stdout | 2025-02-07 05:26:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:04 UTC)" (scheduled at 2025-02-07 05:26:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:26:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:26:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 4, 937429), True, datetime.datetime(2025, 2, 7, 5, 26, 4, 937429), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:26:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:26:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:26:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:26:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:26:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:18.732147+00:00 (in 9.999556 seconds)
notificationworker stdout | 2025-02-07 05:26:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:18 UTC)" (scheduled at 2025-02-07 05:26:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:26:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:26:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 8, 732863), True, datetime.datetime(2025, 2, 7, 5, 26, 8, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:26:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:26:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:26:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:26:08,953 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:26:09,924 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:26:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:26:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:24.423055+00:00 (in 13.998345 seconds)
gcworker stdout | 2025-02-07 05:26:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:40 UTC)" (scheduled at 2025-02-07 05:26:10.424250+00:00)
gcworker stdout | 2025-02-07 05:26:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:26:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905670431, None, 1, 0])
gcworker stdout | 2025-02-07 05:26:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:26:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:26:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:26:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:41.831521+00:00 (in 29.999542 seconds)
autopruneworker stdout | 2025-02-07 05:26:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:41 UTC)" (scheduled at 2025-02-07 05:26:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:26:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902371836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:26:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:26:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:26:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:26:13,891 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:26:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:26:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:29.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 05:26:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:29 UTC)" (scheduled at 2025-02-07 05:26:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:26:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:26:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:26:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:26:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:26:14,235 [238] [DEBUG] [app] Starting request: urn:request:46e65a63-a0f4-49b3-86b7-547736ef955d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:14,235 [235] [DEBUG] [app] Starting request: urn:request:999e0664-16e5-4ac0-a473-5f1d15464088 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:26:14,238 [239] [DEBUG] [app] Starting request: urn:request:e16a03dc-bd42-4b9d-9815-d24388deea4a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:14,238 [244] [DEBUG] [app] Starting request: urn:request:825328d2-df42-4dbd-9b63-9777b4430d36 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:14,238 [239] [DEBUG] [app] Ending request: urn:request:e16a03dc-bd42-4b9d-9815-d24388deea4a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e16a03dc-bd42-4b9d-9815-d24388deea4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:14,238 [244] [DEBUG] [app] Ending request: urn:request:825328d2-df42-4dbd-9b63-9777b4430d36 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:825328d2-df42-4dbd-9b63-9777b4430d36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:26:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:26:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:14,241 [237] [DEBUG] [app] Starting request: urn:request:b1bbaa4e-7771-4601-899c-7f84b753d1f1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:14,241 [238] [DEBUG] [app] Starting request: urn:request:cc28941c-d0c7-4489-a12a-0c0d1a872072 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:14,242 [237] [DEBUG] [app] Ending request: urn:request:b1bbaa4e-7771-4601-899c-7f84b753d1f1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b1bbaa4e-7771-4601-899c-7f84b753d1f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:26:14,242 [238] [DEBUG] [app] Ending request: urn:request:cc28941c-d0c7-4489-a12a-0c0d1a872072 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cc28941c-d0c7-4489-a12a-0c0d1a872072', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:26:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:14,253 [238] [DEBUG] [app] Ending request: urn:request:46e65a63-a0f4-49b3-86b7-547736ef955d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:46e65a63-a0f4-49b3-86b7-547736ef955d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:26:14,254 [235] [DEBUG] [app] Ending request: urn:request:999e0664-16e5-4ac0-a473-5f1d15464088 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:999e0664-16e5-4ac0-a473-5f1d15464088', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:26:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:26:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
builder stdout | 2025-02-07 05:26:14,461 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:26:15,080 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:26:16,436 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:26:17,184 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:26:17,300 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:26:17,700 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:26:17,705 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:26:17,709 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:26:17,714 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:26:17,718 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:26:18,538 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:26:18,721 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:26:18,725 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:26:18,728 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:26:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:26:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:28.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 05:26:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:28 UTC)" (scheduled at 2025-02-07 05:26:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:26:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:26:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 18, 732870), True, datetime.datetime(2025, 2, 7, 5, 26, 18, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:26:18,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:26:18,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:26:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:28 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:18,869 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:26:19,329 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:26:19,376 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:26:19,609 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:26:19,991 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:26:20,126 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:26:20,136 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:26:20,139 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:26:20,142 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:26:20,145 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:26:20,148 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:26:20,151 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:26:20,154 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:26:20,157 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gcworker stdout | 2025-02-07 05:26:20,331 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:26:20,628 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:26:20,798 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:26:22,300 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:26:22,344 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:26:22,815 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:26:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:26:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:03.419110+00:00 (in 39.997710 seconds)
repositorygcworker stdout | 2025-02-07 05:26:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:23 UTC)" (scheduled at 2025-02-07 05:26:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:26:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:26:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:26:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:26:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:40.424250+00:00 (in 16.000740 seconds)
gcworker stdout | 2025-02-07 05:26:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:54 UTC)" (scheduled at 2025-02-07 05:26:24.423055+00:00)
gcworker stdout | 2025-02-07 05:26:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:26:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:26:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:26:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:26:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:38.732147+00:00 (in 9.999510 seconds)
notificationworker stdout | 2025-02-07 05:26:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:38 UTC)" (scheduled at 2025-02-07 05:26:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:26:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:26:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 28, 732938), True, datetime.datetime(2025, 2, 7, 5, 26, 28, 732938), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:26:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:26:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:26:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:26:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:26:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:44.225378+00:00 (in 14.999535 seconds)
quotatotalworker stdout | 2025-02-07 05:26:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:44 UTC)" (scheduled at 2025-02-07 05:26:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:26:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:26:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:26:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:26:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:26:29,235 [237] [DEBUG] [app] Starting request: urn:request:5b5ce4d8-17da-4bf3-b818-ccf037bd6472 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:29,235 [235] [DEBUG] [app] Starting request: urn:request:c1d83a14-0d5b-46f3-a7d0-c16abb693883 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:26:29,238 [244] [DEBUG] [app] Starting request: urn:request:82a3aa79-a608-4baa-bff1-d9a6c1e32939 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:29,238 [245] [DEBUG] [app] Starting request: urn:request:1314252f-fa5f-488c-ad5c-13df57c0dda0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:29,238 [245] [DEBUG] [app] Ending request: urn:request:1314252f-fa5f-488c-ad5c-13df57c0dda0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1314252f-fa5f-488c-ad5c-13df57c0dda0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:29,238 [244] [DEBUG] [app] Ending request: urn:request:82a3aa79-a608-4baa-bff1-d9a6c1e32939 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82a3aa79-a608-4baa-bff1-d9a6c1e32939', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:26:29,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:29,241 [238] [DEBUG] [app] Starting request: urn:request:c4db09cd-811d-4572-b581-160552f6c561 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:29,241 [237] [DEBUG] [app] Starting request: urn:request:36bf80ea-2c82-4811-930f-532c92a02414 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:29,242 [238] [DEBUG] [app] Ending request: urn:request:c4db09cd-811d-4572-b581-160552f6c561 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4db09cd-811d-4572-b581-160552f6c561', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:26:29,242 [237] [DEBUG] [app] Ending request: urn:request:36bf80ea-2c82-4811-930f-532c92a02414 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36bf80ea-2c82-4811-930f-532c92a02414', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:26:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:26:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:29,253 [237] [DEBUG] [app] Ending request: urn:request:5b5ce4d8-17da-4bf3-b818-ccf037bd6472 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5b5ce4d8-17da-4bf3-b818-ccf037bd6472', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:29,254 [235] [DEBUG] [app] Ending request: urn:request:c1d83a14-0d5b-46f3-a7d0-c16abb693883 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c1d83a14-0d5b-46f3-a7d0-c16abb693883', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:26:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:26:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:26:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 05:26:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:26:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:32.625945+00:00 (in 1.004322 seconds)
namespacegcworker stdout | 2025-02-07 05:26:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:31 UTC)" (scheduled at 2025-02-07 05:26:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:26:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:26:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 31, 621898), True, datetime.datetime(2025, 2, 7, 5, 26, 31, 621898), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:26:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:26:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:26:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:26:31,790 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:26:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:26:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:13.624301+00:00 (in 40.997914 seconds)
namespacegcworker stdout | 2025-02-07 05:26:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:32 UTC)" (scheduled at 2025-02-07 05:26:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:26:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:26:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:26:33,528 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:26:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:26:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:03.919286+00:00 (in 29.999523 seconds)
buildlogsarchiver stdout | 2025-02-07 05:26:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:03 UTC)" (scheduled at 2025-02-07 05:26:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:26:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 26, 33, 919948), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:26:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:26:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:26:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:26:34,646 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:56.713547+00:00 (in 18.997192 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:37 UTC)" (scheduled at 2025-02-07 05:26:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:26:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:26:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:48.732147+00:00 (in 9.999549 seconds)
notificationworker stdout | 2025-02-07 05:26:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:48 UTC)" (scheduled at 2025-02-07 05:26:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:26:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:26:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 38, 732900), True, datetime.datetime(2025, 2, 7, 5, 26, 38, 732900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:26:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:26:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:26:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:26:38,969 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:26:39,943 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:26:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:26:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:54.423055+00:00 (in 13.998348 seconds)
gcworker stdout | 2025-02-07 05:26:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:10 UTC)" (scheduled at 2025-02-07 05:26:40.424250+00:00)
gcworker stdout | 2025-02-07 05:26:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:26:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905700431, None, 1, 0])
gcworker stdout | 2025-02-07 05:26:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:26:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:26:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:26:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:11.831521+00:00 (in 29.999580 seconds)
autopruneworker stdout | 2025-02-07 05:26:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:11 UTC)" (scheduled at 2025-02-07 05:26:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:26:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902401836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:26:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:26:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:26:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:26:43,901 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:26:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:26:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:59.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 05:26:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:59 UTC)" (scheduled at 2025-02-07 05:26:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:26:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:26:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:26:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:26:44,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:26:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:26:44,236 [238] [DEBUG] [app] Starting request: urn:request:10432714-e429-40a3-80b4-e7352ce892cf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:44,236 [235] [DEBUG] [app] Starting request: urn:request:9848c2a3-2f87-4568-8319-ee24ac730f53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:44,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:26:44,240 [246] [DEBUG] [app] Starting request: urn:request:e488d496-fdcd-4924-a784-dff8ff69a999 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:44,240 [244] [DEBUG] [app] Starting request: urn:request:6bcc0624-b8bf-41a0-aeaa-415e13c3cac4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:44,241 [246] [DEBUG] [app] Ending request: urn:request:e488d496-fdcd-4924-a784-dff8ff69a999 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e488d496-fdcd-4924-a784-dff8ff69a999', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:44,241 [244] [DEBUG] [app] Ending request: urn:request:6bcc0624-b8bf-41a0-aeaa-415e13c3cac4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6bcc0624-b8bf-41a0-aeaa-415e13c3cac4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:44,241 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:26:44,241 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:26:44,241 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:44,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:44,243 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:44,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:44,245 [235] [DEBUG] [app] Starting request: urn:request:615ede08-9d2d-44e9-b00b-d309628ce324 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:44,245 [238] [DEBUG] [app] Starting request: urn:request:245e2d28-e3c5-4912-ab56-385ff73bd57d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:44,245 [238] [DEBUG] [app] Ending request: urn:request:245e2d28-e3c5-4912-ab56-385ff73bd57d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:245e2d28-e3c5-4912-ab56-385ff73bd57d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:26:44,245 [235] [DEBUG] [app] Ending request: urn:request:615ede08-9d2d-44e9-b00b-d309628ce324 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:615ede08-9d2d-44e9-b00b-d309628ce324', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:26:44,246 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:44,246 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:44,246 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:26:44,246 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:44,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:44,247 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:44,247 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:44,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:44,251 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:44,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:44,251 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:44,256 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:44,256 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:44,257 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:44,258 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:44,259 [238] [DEBUG] [app] Ending request: urn:request:10432714-e429-40a3-80b4-e7352ce892cf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:10432714-e429-40a3-80b4-e7352ce892cf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:44,259 [235] [DEBUG] [app] Ending request: urn:request:9848c2a3-2f87-4568-8319-ee24ac730f53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9848c2a3-2f87-4568-8319-ee24ac730f53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:44,259 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:26:44,259 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.025)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.025)
gunicorn-web stdout | 2025-02-07 05:26:44,260 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:26:44,260 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
builder stdout | 2025-02-07 05:26:44,488 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:26:45,116 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:26:46,469 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:26:47,221 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:26:47,317 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:26:47,709 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:26:47,712 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:26:47,715 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:26:47,721 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:26:47,725 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:26:48,564 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:26:48,729 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
notificationworker stdout | 2025-02-07 05:26:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:26:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:50.814677+00:00 (in 2.082098 seconds)
notificationworker stdout | 2025-02-07 05:26:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:58 UTC)" (scheduled at 2025-02-07 05:26:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:26:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
gunicorn-secscan stdout | 2025-02-07 05:26:48,733 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
notificationworker stdout | 2025-02-07 05:26:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 48, 732855), True, datetime.datetime(2025, 2, 7, 5, 26, 48, 732855), 0, 'notification/%', 50, 1, 0])
gunicorn-secscan stdout | 2025-02-07 05:26:48,736 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
notificationworker stdout | 2025-02-07 05:26:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:26:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:26:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:26:58 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:48,901 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:26:49,343 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:26:49,387 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:26:49,630 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:26:50,019 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:26:50,138 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:26:50,145 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:26:50,148 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:26:50,151 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:26:50,154 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:26:50,156 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:26:50,159 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:26:50,163 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:26:50,166 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:26:50,353 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:26:50,657 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:26:50,804 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
notificationworker stdout | 2025-02-07 05:26:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:26:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:26:58.732147+00:00 (in 7.917031 seconds)
notificationworker stdout | 2025-02-07 05:26:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:50 UTC)" (scheduled at 2025-02-07 05:26:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:26:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:26:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:50 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:26:52,336 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:26:52,380 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:26:52,831 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:26:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:26:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:10.424250+00:00 (in 16.000733 seconds)
gcworker stdout | 2025-02-07 05:26:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:24 UTC)" (scheduled at 2025-02-07 05:26:54.423055+00:00)
gcworker stdout | 2025-02-07 05:26:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:26:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:26:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:26:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:56.118855+00:00 (in 59.999483 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:26:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:56 UTC)" (scheduled at 2025-02-07 05:26:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:26:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:26:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:26:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:37.715878+00:00 (in 41.001820 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:56 UTC)" (scheduled at 2025-02-07 05:26:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:56,715 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:26:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:26:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:26:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:08.732147+00:00 (in 9.999585 seconds)
notificationworker stdout | 2025-02-07 05:26:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:08 UTC)" (scheduled at 2025-02-07 05:26:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:26:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:26:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 26, 58, 732825), True, datetime.datetime(2025, 2, 7, 5, 26, 58, 732825), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:26:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:26:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:26:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:26:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:26:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:14.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 05:26:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:14 UTC)" (scheduled at 2025-02-07 05:26:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:26:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:26:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:26:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:26:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:26:59,235 [237] [DEBUG] [app] Starting request: urn:request:29f41306-068d-4dba-83fb-91e808b83f5c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:59,235 [238] [DEBUG] [app] Starting request: urn:request:1688d795-b606-4a0c-bae6-d34b84dbc992 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:26:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:26:59,238 [246] [DEBUG] [app] Starting request: urn:request:9acb0d0b-d1ba-482d-b80c-d6a5febadc0e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:59,238 [239] [DEBUG] [app] Starting request: urn:request:7b714957-c101-4cf3-a67d-f7c33940026e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:26:59,238 [239] [DEBUG] [app] Ending request: urn:request:7b714957-c101-4cf3-a67d-f7c33940026e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b714957-c101-4cf3-a67d-f7c33940026e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:59,238 [246] [DEBUG] [app] Ending request: urn:request:9acb0d0b-d1ba-482d-b80c-d6a5febadc0e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9acb0d0b-d1ba-482d-b80c-d6a5febadc0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:26:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:26:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:26:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:26:59,241 [238] [DEBUG] [app] Starting request: urn:request:dd557c79-923e-481b-af17-db232412576f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:59,241 [238] [DEBUG] [app] Ending request: urn:request:dd557c79-923e-481b-af17-db232412576f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd557c79-923e-481b-af17-db232412576f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:26:59,241 [236] [DEBUG] [app] Starting request: urn:request:32f0921f-ada6-4d51-92d2-88981e06f285 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:26:59,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:59,241 [236] [DEBUG] [app] Ending request: urn:request:32f0921f-ada6-4d51-92d2-88981e06f285 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32f0921f-ada6-4d51-92d2-88981e06f285', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:26:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:26:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:26:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:26:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:26:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:26:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:26:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:26:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:26:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:26:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:26:59,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:26:59,253 [238] [DEBUG] [app] Ending request: urn:request:1688d795-b606-4a0c-bae6-d34b84dbc992 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1688d795-b606-4a0c-bae6-d34b84dbc992', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:59,253 [237] [DEBUG] [app] Ending request: urn:request:29f41306-068d-4dba-83fb-91e808b83f5c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29f41306-068d-4dba-83fb-91e808b83f5c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:26:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:26:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:26:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:26:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:26:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:26:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:26:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:26:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:04.936688+00:00 (in 4.998315 seconds)
exportactionlogsworker stdout | 2025-02-07 05:26:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:59 UTC)" (scheduled at 2025-02-07 05:26:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:26:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:26:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:27:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:27:01,817 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:27:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:27:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:23.420886+00:00 (in 20.001288 seconds)
repositorygcworker stdout | 2025-02-07 05:27:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:03 UTC)" (scheduled at 2025-02-07 05:27:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:27:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:27:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 3, 419881), True, datetime.datetime(2025, 2, 7, 5, 27, 3, 419881), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:27:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:27:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:27:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:27:03,545 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:27:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:27:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:33.919286+00:00 (in 29.999500 seconds)
buildlogsarchiver stdout | 2025-02-07 05:27:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:33 UTC)" (scheduled at 2025-02-07 05:27:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:27:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 27, 3, 920053), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:27:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:27:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:27:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:27:04,682 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:27:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:27:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:59.937899+00:00 (in 55.000758 seconds)
exportactionlogsworker stdout | 2025-02-07 05:27:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:04 UTC)" (scheduled at 2025-02-07 05:27:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:27:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:27:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 4, 937447), True, datetime.datetime(2025, 2, 7, 5, 27, 4, 937447), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:27:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:27:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:27:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:27:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:18.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:27:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:18 UTC)" (scheduled at 2025-02-07 05:27:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:27:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:27:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 8, 732849), True, datetime.datetime(2025, 2, 7, 5, 27, 8, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:27:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:27:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:27:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:27:09,005 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:27:09,968 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:27:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:27:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:24.423055+00:00 (in 13.998348 seconds)
gcworker stdout | 2025-02-07 05:27:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:40 UTC)" (scheduled at 2025-02-07 05:27:10.424250+00:00)
gcworker stdout | 2025-02-07 05:27:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:27:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905730431, None, 1, 0])
gcworker stdout | 2025-02-07 05:27:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:27:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:27:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:27:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:41.831521+00:00 (in 29.999551 seconds)
autopruneworker stdout | 2025-02-07 05:27:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:41 UTC)" (scheduled at 2025-02-07 05:27:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:27:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902431836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:27:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:27:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:27:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:41 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:27:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:27:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:31.621131+00:00 (in 17.996398 seconds)
namespacegcworker stdout | 2025-02-07 05:27:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:32:13 UTC)" (scheduled at 2025-02-07 05:27:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:27:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 27, 13, 624976), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:27:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 27, 13, 624976), True, datetime.datetime(2025, 2, 7, 5, 27, 13, 624976), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:27:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 27, 13, 624976), True, datetime.datetime(2025, 2, 7, 5, 27, 13, 624976), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 27, 13, 624976), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:27:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:27:13,634 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:32:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:27:13,934 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:27:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:27:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:29.225378+00:00 (in 14.999528 seconds)
quotatotalworker stdout | 2025-02-07 05:27:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:29 UTC)" (scheduled at 2025-02-07 05:27:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:27:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:27:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:27:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:27:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:27:14,235 [238] [DEBUG] [app] Starting request: urn:request:62f1a0ea-6643-44e0-8263-1ae409cdfbed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:14,235 [235] [DEBUG] [app] Starting request: urn:request:4290802c-d6d7-42d8-87ea-4bf9774ae47e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:14,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:27:14,239 [239] [DEBUG] [app] Starting request: urn:request:fb7392ac-f3ac-418a-897a-77e9d9d27672 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:14,239 [245] [DEBUG] [app] Starting request: urn:request:2a8631ae-0c1c-4259-be46-cee344e351eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:14,239 [239] [DEBUG] [app] Ending request: urn:request:fb7392ac-f3ac-418a-897a-77e9d9d27672 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fb7392ac-f3ac-418a-897a-77e9d9d27672', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:27:14,240 [245] [DEBUG] [app] Ending request: urn:request:2a8631ae-0c1c-4259-be46-cee344e351eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a8631ae-0c1c-4259-be46-cee344e351eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:27:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:27:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:27:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:14,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:14,243 [237] [DEBUG] [app] Starting request: urn:request:610ef2b1-3627-4f7d-9faa-c1cdd7ad8262 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:27:14,244 [236] [DEBUG] [app] Starting request: urn:request:16ce6cec-303f-4096-8839-46a502619030 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:27:14,244 [237] [DEBUG] [app] Ending request: urn:request:610ef2b1-3627-4f7d-9faa-c1cdd7ad8262 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:610ef2b1-3627-4f7d-9faa-c1cdd7ad8262', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:27:14,244 [236] [DEBUG] [app] Ending request: urn:request:16ce6cec-303f-4096-8839-46a502619030 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16ce6cec-303f-4096-8839-46a502619030', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:27:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:27:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:27:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:14,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:14,256 [235] [DEBUG] [app] Ending request: urn:request:4290802c-d6d7-42d8-87ea-4bf9774ae47e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4290802c-d6d7-42d8-87ea-4bf9774ae47e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:14,256 [238] [DEBUG] [app] Ending request: urn:request:62f1a0ea-6643-44e0-8263-1ae409cdfbed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:62f1a0ea-6643-44e0-8263-1ae409cdfbed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:27:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.023 118 0.023)
gunicorn-web stdout | 2025-02-07 05:27:14,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:27:14,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
builder stdout | 2025-02-07 05:27:14,511 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:27:15,126 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:27:16,498 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:27:17,257 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:27:17,353 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:27:17,717 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:27:17,720 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:27:17,722 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:27:17,728 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:27:17,732 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:27:18,600 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
notificationworker stdout | 2025-02-07 05:27:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:28.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:27:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:28 UTC)" (scheduled at 2025-02-07 05:27:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:27:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:27:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 18, 732850), True, datetime.datetime(2025, 2, 7, 5, 27, 18, 732850), 0, 'notification/%', 50, 1, 0])
gunicorn-secscan stdout | 2025-02-07 05:27:18,736 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:27:18,739 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
notificationworker stdout | 2025-02-07 05:27:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:27:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:27:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:27:18,742 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:18,936 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:27:19,377 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:27:19,418 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:27:19,667 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:27:20,056 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:27:20,149 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:27:20,155 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:27:20,158 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:27:20,162 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:27:20,165 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:27:20,169 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:27:20,172 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:27:20,175 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:27:20,178 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:27:20,389 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:27:20,694 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:27:20,832 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:27:22,343 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:27:22,391 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:27:22,866 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:27:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:27:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:03.419110+00:00 (in 39.997757 seconds)
repositorygcworker stdout | 2025-02-07 05:27:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:23 UTC)" (scheduled at 2025-02-07 05:27:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:27:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:27:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:27:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:27:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:40.424250+00:00 (in 16.000749 seconds)
gcworker stdout | 2025-02-07 05:27:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:54 UTC)" (scheduled at 2025-02-07 05:27:24.423055+00:00)
gcworker stdout | 2025-02-07 05:27:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:27:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:27:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:27:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:33.814166+00:00 (in 5.081571 seconds)
notificationworker stdout | 2025-02-07 05:27:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:38 UTC)" (scheduled at 2025-02-07 05:27:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:27:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:27:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 28, 732870), True, datetime.datetime(2025, 2, 7, 5, 27, 28, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:27:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:27:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:27:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:27:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:27:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:44.225378+00:00 (in 14.999546 seconds)
quotatotalworker stdout | 2025-02-07 05:27:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:44 UTC)" (scheduled at 2025-02-07 05:27:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:27:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:27:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:27:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:27:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:27:29,235 [235] [DEBUG] [app] Starting request: urn:request:742253e4-7732-4f73-a33c-03bb64dcc97c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:29,235 [238] [DEBUG] [app] Starting request: urn:request:9f6aed92-2c96-4bcc-9886-6f459d752456 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:27:29,238 [239] [DEBUG] [app] Starting request: urn:request:93347281-614f-4fbd-8cb4-8916974c760b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:29,238 [242] [DEBUG] [app] Starting request: urn:request:88d31eec-07d1-424c-9b8a-4846b277a1da (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:29,238 [239] [DEBUG] [app] Ending request: urn:request:93347281-614f-4fbd-8cb4-8916974c760b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:93347281-614f-4fbd-8cb4-8916974c760b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:27:29,238 [242] [DEBUG] [app] Ending request: urn:request:88d31eec-07d1-424c-9b8a-4846b277a1da (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:88d31eec-07d1-424c-9b8a-4846b277a1da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:27:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:27:29,239 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:29,241 [238] [DEBUG] [app] Starting request: urn:request:7b3d156c-4d35-4533-bcef-4357d82c489a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:27:29,241 [238] [DEBUG] [app] Ending request: urn:request:7b3d156c-4d35-4533-bcef-4357d82c489a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7b3d156c-4d35-4533-bcef-4357d82c489a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:27:29,241 [237] [DEBUG] [app] Starting request: urn:request:8f911e04-9915-4400-af02-c18fa62da793 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:27:29,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:29,242 [237] [DEBUG] [app] Ending request: urn:request:8f911e04-9915-4400-af02-c18fa62da793 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8f911e04-9915-4400-af02-c18fa62da793', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:27:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:27:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:27:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:29,253 [238] [DEBUG] [app] Ending request: urn:request:9f6aed92-2c96-4bcc-9886-6f459d752456 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f6aed92-2c96-4bcc-9886-6f459d752456', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:29,253 [235] [DEBUG] [app] Ending request: urn:request:742253e4-7732-4f73-a33c-03bb64dcc97c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:742253e4-7732-4f73-a33c-03bb64dcc97c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:27:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:27:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:27:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 05:27:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:27:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:32.625945+00:00 (in 1.004350 seconds)
namespacegcworker stdout | 2025-02-07 05:27:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:31 UTC)" (scheduled at 2025-02-07 05:27:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:27:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:27:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 31, 621868), True, datetime.datetime(2025, 2, 7, 5, 27, 31, 621868), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:27:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:27:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:27:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:27:31,853 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:27:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:27:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:31.621131+00:00 (in 58.994726 seconds)
namespacegcworker stdout | 2025-02-07 05:27:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:32 UTC)" (scheduled at 2025-02-07 05:27:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:27:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:27:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:27:33,555 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 05:27:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:38.732147+00:00 (in 4.917529 seconds)
notificationworker stdout | 2025-02-07 05:27:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:32:33 UTC)" (scheduled at 2025-02-07 05:27:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:27:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 27, 33, 814864), 'notification/%'])
notificationworker stdout | 2025-02-07 05:27:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 27, 33, 814864), True, datetime.datetime(2025, 2, 7, 5, 27, 33, 814864), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:27:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 27, 33, 814864), True, datetime.datetime(2025, 2, 7, 5, 27, 33, 814864), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 27, 33, 814864), 'notification/%'])
notificationworker stdout | 2025-02-07 05:27:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:27:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:32:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:27:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:27:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:03.919286+00:00 (in 29.999486 seconds)
buildlogsarchiver stdout | 2025-02-07 05:27:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:03 UTC)" (scheduled at 2025-02-07 05:27:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:27:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 27, 33, 920090), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:27:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:27:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:27:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:27:34,711 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:56.713547+00:00 (in 18.997193 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:37 UTC)" (scheduled at 2025-02-07 05:27:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:27:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:48.732147+00:00 (in 9.999593 seconds)
notificationworker stdout | 2025-02-07 05:27:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:48 UTC)" (scheduled at 2025-02-07 05:27:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:27:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:27:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 38, 732830), True, datetime.datetime(2025, 2, 7, 5, 27, 38, 732830), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:27:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:27:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:27:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:27:39,041 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:27:39,978 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:27:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:27:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:54.423055+00:00 (in 13.998346 seconds)
gcworker stdout | 2025-02-07 05:27:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:10 UTC)" (scheduled at 2025-02-07 05:27:40.424250+00:00)
gcworker stdout | 2025-02-07 05:27:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:27:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905760431, None, 1, 0])
gcworker stdout | 2025-02-07 05:27:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:27:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:27:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:27:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:11.831521+00:00 (in 29.999565 seconds)
autopruneworker stdout | 2025-02-07 05:27:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:11 UTC)" (scheduled at 2025-02-07 05:27:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:27:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902461835, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:27:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:27:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:27:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:27:43,971 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:27:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:27:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:59.225378+00:00 (in 14.999542 seconds)
quotatotalworker stdout | 2025-02-07 05:27:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:59 UTC)" (scheduled at 2025-02-07 05:27:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:27:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:27:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:27:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:27:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:27:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:27:44,235 [236] [DEBUG] [app] Starting request: urn:request:0044d9c6-8b42-4cf6-ad84-a935f7c65a08 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:44,235 [235] [DEBUG] [app] Starting request: urn:request:74ba1aed-04ec-4fed-805f-38690d691e88 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:27:44,239 [239] [DEBUG] [app] Starting request: urn:request:125b0c48-9395-445a-a0b5-4cb2c8e79793 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:44,240 [239] [DEBUG] [app] Ending request: urn:request:125b0c48-9395-445a-a0b5-4cb2c8e79793 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:125b0c48-9395-445a-a0b5-4cb2c8e79793', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:27:44,239 [246] [DEBUG] [app] Starting request: urn:request:8344fe92-0252-4636-a387-3b698905a0a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:27:44,240 [246] [DEBUG] [app] Ending request: urn:request:8344fe92-0252-4636-a387-3b698905a0a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8344fe92-0252-4636-a387-3b698905a0a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:27:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:27:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:27:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:44,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:44,243 [238] [DEBUG] [app] Starting request: urn:request:b0bc7350-8e77-4823-bf29-ea854d114d0e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:27:44,243 [236] [DEBUG] [app] Starting request: urn:request:a196493b-7073-459b-a796-1f5bc08c3f52 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:27:44,243 [238] [DEBUG] [app] Ending request: urn:request:b0bc7350-8e77-4823-bf29-ea854d114d0e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b0bc7350-8e77-4823-bf29-ea854d114d0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:27:44,243 [236] [DEBUG] [app] Ending request: urn:request:a196493b-7073-459b-a796-1f5bc08c3f52 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a196493b-7073-459b-a796-1f5bc08c3f52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:27:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:27:44,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:27:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:27:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:44,254 [236] [DEBUG] [app] Ending request: urn:request:0044d9c6-8b42-4cf6-ad84-a935f7c65a08 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0044d9c6-8b42-4cf6-ad84-a935f7c65a08', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:44,254 [235] [DEBUG] [app] Ending request: urn:request:74ba1aed-04ec-4fed-805f-38690d691e88 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:74ba1aed-04ec-4fed-805f-38690d691e88', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:27:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:27:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:27:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
builder stdout | 2025-02-07 05:27:44,547 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:27:45,150 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:27:46,535 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:27:47,293 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:27:47,389 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:27:47,726 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:27:47,730 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:27:47,733 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:27:47,736 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:27:47,739 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:27:48,636 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
notificationworker stdout | 2025-02-07 05:27:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:50.814677+00:00 (in 2.082073 seconds)
notificationworker stdout | 2025-02-07 05:27:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:58 UTC)" (scheduled at 2025-02-07 05:27:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:27:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:27:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 48, 732886), True, datetime.datetime(2025, 2, 7, 5, 27, 48, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:27:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:27:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:27:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:27:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:27:48,745 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:27:48,748 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:27:48,751 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:48,972 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:27:49,385 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:27:49,448 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:27:49,704 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:27:50,079 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:27:50,161 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:27:50,165 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:27:50,167 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:27:50,170 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:27:50,172 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:27:50,175 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:27:50,178 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:27:50,186 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:27:50,189 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:27:50,422 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:27:50,730 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
notificationworker stdout | 2025-02-07 05:27:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:27:58.732147+00:00 (in 7.917043 seconds)
notificationworker stdout | 2025-02-07 05:27:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:50 UTC)" (scheduled at 2025-02-07 05:27:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:27:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:27:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:50 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 05:27:50,839 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:27:52,378 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:27:52,417 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:27:52,902 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:27:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:27:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 05:27:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:24 UTC)" (scheduled at 2025-02-07 05:27:54.423055+00:00)
gcworker stdout | 2025-02-07 05:27:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:27:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:27:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:27:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:56.118855+00:00 (in 59.999487 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:27:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:56 UTC)" (scheduled at 2025-02-07 05:27:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:27:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:27:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:27:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:37.715878+00:00 (in 41.001858 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:56 UTC)" (scheduled at 2025-02-07 05:27:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:27:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:27:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:27:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:08.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 05:27:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:08 UTC)" (scheduled at 2025-02-07 05:27:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:27:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:27:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 27, 58, 732917), True, datetime.datetime(2025, 2, 7, 5, 27, 58, 732917), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:27:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:27:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:27:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:27:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:27:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:14.225378+00:00 (in 14.999481 seconds)
quotatotalworker stdout | 2025-02-07 05:27:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:14 UTC)" (scheduled at 2025-02-07 05:27:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:27:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:27:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:27:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:27:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:27:59,235 [235] [DEBUG] [app] Starting request: urn:request:986c8db9-afda-444e-a2f6-2b03666325d7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:59,235 [238] [DEBUG] [app] Starting request: urn:request:95b0cdaf-f50d-42ed-bdaa-3a5bc11fe0f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:27:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:27:59,238 [239] [DEBUG] [app] Starting request: urn:request:8f535f6a-fd4d-420d-a0e9-60dc323345d0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:59,238 [245] [DEBUG] [app] Starting request: urn:request:614ca633-d15a-4da8-b0e0-2a60c051d0d6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:27:59,239 [239] [DEBUG] [app] Ending request: urn:request:8f535f6a-fd4d-420d-a0e9-60dc323345d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8f535f6a-fd4d-420d-a0e9-60dc323345d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:27:59,239 [245] [DEBUG] [app] Ending request: urn:request:614ca633-d15a-4da8-b0e0-2a60c051d0d6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:614ca633-d15a-4da8-b0e0-2a60c051d0d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:27:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:27:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:27:59,242 [235] [DEBUG] [app] Starting request: urn:request:25f55b82-6af4-4e23-acf8-77dafa329a9e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:27:59,242 [235] [DEBUG] [app] Ending request: urn:request:25f55b82-6af4-4e23-acf8-77dafa329a9e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25f55b82-6af4-4e23-acf8-77dafa329a9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:27:59,242 [237] [DEBUG] [app] Starting request: urn:request:56bd28f1-ae18-4efd-9429-008d2be91e1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:27:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:59,242 [237] [DEBUG] [app] Ending request: urn:request:56bd28f1-ae18-4efd-9429-008d2be91e1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56bd28f1-ae18-4efd-9429-008d2be91e1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:27:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:27:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:27:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:27:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:27:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:27:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:27:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:27:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:27:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:27:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:27:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:27:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:27:59,254 [238] [DEBUG] [app] Ending request: urn:request:95b0cdaf-f50d-42ed-bdaa-3a5bc11fe0f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:95b0cdaf-f50d-42ed-bdaa-3a5bc11fe0f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:59,254 [235] [DEBUG] [app] Ending request: urn:request:986c8db9-afda-444e-a2f6-2b03666325d7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:986c8db9-afda-444e-a2f6-2b03666325d7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:27:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:27:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:27:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:27:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:27:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:27:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:27:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:27:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:04.936688+00:00 (in 4.998265 seconds)
exportactionlogsworker stdout | 2025-02-07 05:27:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:59 UTC)" (scheduled at 2025-02-07 05:27:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:27:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:27:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:28:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:28:01,890 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:28:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:28:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:23.420886+00:00 (in 20.001292 seconds)
repositorygcworker stdout | 2025-02-07 05:28:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:03 UTC)" (scheduled at 2025-02-07 05:28:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:28:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:28:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 3, 419924), True, datetime.datetime(2025, 2, 7, 5, 28, 3, 419924), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:28:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:28:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:28:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:28:03,570 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:28:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:28:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:33.919286+00:00 (in 29.999501 seconds)
buildlogsarchiver stdout | 2025-02-07 05:28:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:33 UTC)" (scheduled at 2025-02-07 05:28:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:28:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 28, 3, 919971), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:28:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:28:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:28:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:28:04,742 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:28:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:28:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:59.937899+00:00 (in 55.000773 seconds)
exportactionlogsworker stdout | 2025-02-07 05:28:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:04 UTC)" (scheduled at 2025-02-07 05:28:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:28:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:28:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 4, 937428), True, datetime.datetime(2025, 2, 7, 5, 28, 4, 937428), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:28:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:28:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:28:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:28:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:28:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:18.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 05:28:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:18 UTC)" (scheduled at 2025-02-07 05:28:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:28:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:28:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 8, 732929), True, datetime.datetime(2025, 2, 7, 5, 28, 8, 732929), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:28:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:28:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:28:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:28:09,077 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:28:10,014 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:28:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:28:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:24.423055+00:00 (in 13.998324 seconds)
gcworker stdout | 2025-02-07 05:28:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:40 UTC)" (scheduled at 2025-02-07 05:28:10.424250+00:00)
gcworker stdout | 2025-02-07 05:28:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:28:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905790430, None, 1, 0])
gcworker stdout | 2025-02-07 05:28:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:28:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:28:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:28:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:41.831521+00:00 (in 29.999515 seconds)
autopruneworker stdout | 2025-02-07 05:28:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:41 UTC)" (scheduled at 2025-02-07 05:28:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:28:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902491836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:28:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:28:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:28:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:28:14,005 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:28:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:28:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:29.225378+00:00 (in 14.999453 seconds)
quotatotalworker stdout | 2025-02-07 05:28:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:29 UTC)" (scheduled at 2025-02-07 05:28:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:28:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:28:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:28:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:28:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:28:14,235 [235] [DEBUG] [app] Starting request: urn:request:7afd552a-35c7-4527-a401-d75ab46540b8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:14,235 [238] [DEBUG] [app] Starting request: urn:request:0b3078ef-d4eb-4e1e-a3fd-fba627367768 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:28:14,238 [246] [DEBUG] [app] Starting request: urn:request:e1dc9d8c-9082-4492-ae93-889392a08ced (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:14,238 [239] [DEBUG] [app] Starting request: urn:request:e85d57d6-3640-4d9c-910c-c31eb23353d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:14,239 [239] [DEBUG] [app] Ending request: urn:request:e85d57d6-3640-4d9c-910c-c31eb23353d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e85d57d6-3640-4d9c-910c-c31eb23353d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:28:14,239 [246] [DEBUG] [app] Ending request: urn:request:e1dc9d8c-9082-4492-ae93-889392a08ced (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e1dc9d8c-9082-4492-ae93-889392a08ced', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:28:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:28:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:14,242 [238] [DEBUG] [app] Starting request: urn:request:de5bc76b-7924-4204-9e2a-98e00043c4c5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:14,242 [236] [DEBUG] [app] Starting request: urn:request:0672cb86-64e8-4086-9c31-3c92cdea17c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:14,242 [238] [DEBUG] [app] Ending request: urn:request:de5bc76b-7924-4204-9e2a-98e00043c4c5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:de5bc76b-7924-4204-9e2a-98e00043c4c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:28:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:14,242 [236] [DEBUG] [app] Ending request: urn:request:0672cb86-64e8-4086-9c31-3c92cdea17c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0672cb86-64e8-4086-9c31-3c92cdea17c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:28:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:28:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:28:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:28:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:14,254 [238] [DEBUG] [app] Ending request: urn:request:0b3078ef-d4eb-4e1e-a3fd-fba627367768 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b3078ef-d4eb-4e1e-a3fd-fba627367768', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:28:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:28:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:28:14,255 [235] [DEBUG] [app] Ending request: urn:request:7afd552a-35c7-4527-a401-d75ab46540b8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7afd552a-35c7-4527-a401-d75ab46540b8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:28:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:28:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
builder stdout | 2025-02-07 05:28:14,579 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:28:15,162 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:28:16,559 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:28:17,330 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:28:17,425 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:28:17,735 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:28:17,738 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:28:17,740 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:28:17,743 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:28:17,747 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:28:18,666 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
notificationworker stdout | 2025-02-07 05:28:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:28:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:28.732147+00:00 (in 9.999551 seconds)
notificationworker stdout | 2025-02-07 05:28:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:28 UTC)" (scheduled at 2025-02-07 05:28:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:28:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:28:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 18, 732883), True, datetime.datetime(2025, 2, 7, 5, 28, 18, 732883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:28:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:28:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:28:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:28:18,754 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:28:18,757 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:28:18,760 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:19,009 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:28:19,395 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:28:19,470 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:28:19,740 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:28:20,115 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:28:20,170 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:28:20,173 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:28:20,177 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:28:20,180 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:28:20,183 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:28:20,186 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:28:20,190 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:28:20,196 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:28:20,200 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:28:20,450 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:28:20,744 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:28:20,858 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:28:22,411 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:28:22,453 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:28:22,925 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:28:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:28:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:32.420707+00:00 (in 8.999301 seconds)
repositorygcworker stdout | 2025-02-07 05:28:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:23 UTC)" (scheduled at 2025-02-07 05:28:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:28:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:28:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:28:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:28:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:40.424250+00:00 (in 16.000710 seconds)
gcworker stdout | 2025-02-07 05:28:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:54 UTC)" (scheduled at 2025-02-07 05:28:24.423055+00:00)
gcworker stdout | 2025-02-07 05:28:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:28:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:28:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:28:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:28:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:38.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 05:28:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:38 UTC)" (scheduled at 2025-02-07 05:28:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:28:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:28:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 28, 732903), True, datetime.datetime(2025, 2, 7, 5, 28, 28, 732903), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:28:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:28:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:28:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:28:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:28:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:44.225378+00:00 (in 14.999523 seconds)
quotatotalworker stdout | 2025-02-07 05:28:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:44 UTC)" (scheduled at 2025-02-07 05:28:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:28:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:28:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:28:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:28:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:28:29,235 [236] [DEBUG] [app] Starting request: urn:request:94044b04-7226-483a-8c40-5a53fd078e09 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:29,235 [238] [DEBUG] [app] Starting request: urn:request:5d3debe6-57c5-4b09-b9a5-82bfe2d1f246 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:28:29,238 [239] [DEBUG] [app] Starting request: urn:request:b6c275ef-ce02-4134-b646-fb754381001f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:29,238 [243] [DEBUG] [app] Starting request: urn:request:351f6196-82c5-4a09-99ef-2f24be267243 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:29,239 [239] [DEBUG] [app] Ending request: urn:request:b6c275ef-ce02-4134-b646-fb754381001f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b6c275ef-ce02-4134-b646-fb754381001f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:28:29,239 [243] [DEBUG] [app] Ending request: urn:request:351f6196-82c5-4a09-99ef-2f24be267243 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:351f6196-82c5-4a09-99ef-2f24be267243', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:28:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:28:29,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:28:29,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:29,243 [235] [DEBUG] [app] Starting request: urn:request:fed4e863-2233-41e8-a3db-3f848e9c704c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:29,243 [237] [DEBUG] [app] Starting request: urn:request:96798dc3-0cff-4e01-b11b-e0058357f585 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:29,243 [237] [DEBUG] [app] Ending request: urn:request:96798dc3-0cff-4e01-b11b-e0058357f585 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:96798dc3-0cff-4e01-b11b-e0058357f585', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:28:29,243 [235] [DEBUG] [app] Ending request: urn:request:fed4e863-2233-41e8-a3db-3f848e9c704c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fed4e863-2233-41e8-a3db-3f848e9c704c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:28:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:28:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:28:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:28:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:29,255 [238] [DEBUG] [app] Ending request: urn:request:5d3debe6-57c5-4b09-b9a5-82bfe2d1f246 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d3debe6-57c5-4b09-b9a5-82bfe2d1f246', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:28:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:28:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:28:29,255 [236] [DEBUG] [app] Ending request: urn:request:94044b04-7226-483a-8c40-5a53fd078e09 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94044b04-7226-483a-8c40-5a53fd078e09', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:28:29,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:28:29,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 05:28:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:28:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:32.625945+00:00 (in 1.004350 seconds)
namespacegcworker stdout | 2025-02-07 05:28:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:31 UTC)" (scheduled at 2025-02-07 05:28:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:28:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:28:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 31, 621861), True, datetime.datetime(2025, 2, 7, 5, 28, 31, 621861), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:28:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:28:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:28:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:28:31,927 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:28:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:28:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:03.419110+00:00 (in 30.997934 seconds)
repositorygcworker stdout | 2025-02-07 05:28:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:33:32 UTC)" (scheduled at 2025-02-07 05:28:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:28:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 28, 32, 421449), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:28:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 28, 32, 421449), True, datetime.datetime(2025, 2, 7, 5, 28, 32, 421449), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:28:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 28, 32, 421449), True, datetime.datetime(2025, 2, 7, 5, 28, 32, 421449), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 28, 32, 421449), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:28:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:28:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:33:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:28:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:28:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:31.621131+00:00 (in 58.994732 seconds)
namespacegcworker stdout | 2025-02-07 05:28:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:32 UTC)" (scheduled at 2025-02-07 05:28:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:28:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:28:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:28:33,580 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:28:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:28:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:03.919286+00:00 (in 29.999536 seconds)
buildlogsarchiver stdout | 2025-02-07 05:28:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:03 UTC)" (scheduled at 2025-02-07 05:28:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:28:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 28, 33, 919994), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:28:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:28:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:28:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:28:34,774 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:56.713547+00:00 (in 18.997198 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:37 UTC)" (scheduled at 2025-02-07 05:28:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:28:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:28:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:48.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 05:28:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:48 UTC)" (scheduled at 2025-02-07 05:28:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:28:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:28:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 38, 732858), True, datetime.datetime(2025, 2, 7, 5, 28, 38, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:28:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:28:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:28:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:28:39,098 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:28:40,051 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:28:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:28:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:54.423055+00:00 (in 13.998328 seconds)
gcworker stdout | 2025-02-07 05:28:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:10 UTC)" (scheduled at 2025-02-07 05:28:40.424250+00:00)
gcworker stdout | 2025-02-07 05:28:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:28:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905820431, None, 1, 0])
gcworker stdout | 2025-02-07 05:28:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:28:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:28:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:28:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:11.831521+00:00 (in 29.999505 seconds)
autopruneworker stdout | 2025-02-07 05:28:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:11 UTC)" (scheduled at 2025-02-07 05:28:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:28:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902521836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:28:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:28:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:28:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:28:44,019 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:28:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:28:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:59.225378+00:00 (in 14.999584 seconds)
quotatotalworker stdout | 2025-02-07 05:28:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:59 UTC)" (scheduled at 2025-02-07 05:28:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:28:44,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:28:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:28:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:28:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:28:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:28:44,235 [237] [DEBUG] [app] Starting request: urn:request:824ce8d5-30c0-49b1-9351-296c40c9abb1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:44,235 [235] [DEBUG] [app] Starting request: urn:request:76af7ec4-7050-4213-ab10-6f1f5c06b18d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:28:44,239 [246] [DEBUG] [app] Starting request: urn:request:1f631082-c15c-4eeb-81a5-2cd7c0f117b6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:44,239 [239] [DEBUG] [app] Starting request: urn:request:54c338b4-03fe-4c1e-8716-d939ac16a42b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:44,239 [239] [DEBUG] [app] Ending request: urn:request:54c338b4-03fe-4c1e-8716-d939ac16a42b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:54c338b4-03fe-4c1e-8716-d939ac16a42b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:28:44,239 [246] [DEBUG] [app] Ending request: urn:request:1f631082-c15c-4eeb-81a5-2cd7c0f117b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1f631082-c15c-4eeb-81a5-2cd7c0f117b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:28:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:28:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:44,242 [235] [DEBUG] [app] Starting request: urn:request:bbb760fb-f57a-4bb1-b348-4c51c3b9eccd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:44,242 [235] [DEBUG] [app] Ending request: urn:request:bbb760fb-f57a-4bb1-b348-4c51c3b9eccd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bbb760fb-f57a-4bb1-b348-4c51c3b9eccd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:28:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:44,243 [238] [DEBUG] [app] Starting request: urn:request:56f1dd83-6320-4153-82b3-9cd829579513 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:44,243 [238] [DEBUG] [app] Ending request: urn:request:56f1dd83-6320-4153-82b3-9cd829579513 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56f1dd83-6320-4153-82b3-9cd829579513', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:28:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:28:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:28:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:44,254 [235] [DEBUG] [app] Ending request: urn:request:76af7ec4-7050-4213-ab10-6f1f5c06b18d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76af7ec4-7050-4213-ab10-6f1f5c06b18d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:28:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:28:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:28:44,254 [237] [DEBUG] [app] Ending request: urn:request:824ce8d5-30c0-49b1-9351-296c40c9abb1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:824ce8d5-30c0-49b1-9351-296c40c9abb1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:28:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:28:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
builder stdout | 2025-02-07 05:28:44,615 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:28:45,198 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:28:46,596 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:28:47,366 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:28:47,446 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:28:47,743 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:28:47,746 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:28:47,750 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:28:47,752 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:28:47,755 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:28:48,703 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
notificationworker stdout | 2025-02-07 05:28:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:28:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:50.814677+00:00 (in 2.082089 seconds)
notificationworker stdout | 2025-02-07 05:28:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:58 UTC)" (scheduled at 2025-02-07 05:28:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:28:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:28:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 48, 732879), True, datetime.datetime(2025, 2, 7, 5, 28, 48, 732879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:28:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:28:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:28:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:28:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:28:48,761 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:28:48,765 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:28:48,768 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:49,045 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:28:49,404 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:28:49,487 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:28:49,777 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:28:50,140 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:28:50,180 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:28:50,182 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:28:50,185 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:28:50,189 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:28:50,193 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:28:50,196 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:28:50,200 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:28:50,205 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:28:50,207 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gcworker stdout | 2025-02-07 05:28:50,465 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:28:50,777 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
notificationworker stdout | 2025-02-07 05:28:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:28:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:28:58.732147+00:00 (in 7.917033 seconds)
notificationworker stdout | 2025-02-07 05:28:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:50 UTC)" (scheduled at 2025-02-07 05:28:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:28:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:28:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:50 UTC)" executed successfully
manifestbackfillworker stdout | 2025-02-07 05:28:50,876 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:28:52,440 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:28:52,488 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:28:52,954 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:28:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:28:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:10.424250+00:00 (in 16.000724 seconds)
gcworker stdout | 2025-02-07 05:28:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:24 UTC)" (scheduled at 2025-02-07 05:28:54.423055+00:00)
gcworker stdout | 2025-02-07 05:28:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:28:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:28:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:28:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:56.118855+00:00 (in 59.999540 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:28:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:56 UTC)" (scheduled at 2025-02-07 05:28:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:28:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:28:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:28:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:37.715878+00:00 (in 41.001890 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:56 UTC)" (scheduled at 2025-02-07 05:28:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:28:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:28:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:28:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:08.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 05:28:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:08 UTC)" (scheduled at 2025-02-07 05:28:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:28:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:28:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 28, 58, 732867), True, datetime.datetime(2025, 2, 7, 5, 28, 58, 732867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:28:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:28:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:28:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:28:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:28:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:14.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 05:28:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:14 UTC)" (scheduled at 2025-02-07 05:28:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:28:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:28:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:28:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:28:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:28:59,236 [236] [DEBUG] [app] Starting request: urn:request:2e5ed6b4-970c-4513-acad-4d3010ac2bc0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:59,236 [235] [DEBUG] [app] Starting request: urn:request:72a9aa7b-2e8f-4905-8655-179fa9c9ac29 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:28:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:28:59,238 [246] [DEBUG] [app] Starting request: urn:request:9c062054-1479-4855-8de5-271f56606a42 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:59,238 [239] [DEBUG] [app] Starting request: urn:request:53ca0f33-c86f-49e8-8494-e528e70fc8aa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:28:59,239 [246] [DEBUG] [app] Ending request: urn:request:9c062054-1479-4855-8de5-271f56606a42 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c062054-1479-4855-8de5-271f56606a42', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:28:59,239 [239] [DEBUG] [app] Ending request: urn:request:53ca0f33-c86f-49e8-8494-e528e70fc8aa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:53ca0f33-c86f-49e8-8494-e528e70fc8aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:28:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:28:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:28:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:28:59,241 [235] [DEBUG] [app] Starting request: urn:request:10593f8c-f36e-44d3-844f-c475cada1ff4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:59,242 [238] [DEBUG] [app] Starting request: urn:request:02926690-813d-41a5-a485-ecf6cf9a2193 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:28:59,242 [235] [DEBUG] [app] Ending request: urn:request:10593f8c-f36e-44d3-844f-c475cada1ff4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:10593f8c-f36e-44d3-844f-c475cada1ff4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:28:59,242 [238] [DEBUG] [app] Ending request: urn:request:02926690-813d-41a5-a485-ecf6cf9a2193 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02926690-813d-41a5-a485-ecf6cf9a2193', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:28:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:28:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:28:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:28:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:28:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:28:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:28:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:28:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:28:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:28:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:59,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:28:59,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:28:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:59,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:28:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:28:59,253 [235] [DEBUG] [app] Ending request: urn:request:72a9aa7b-2e8f-4905-8655-179fa9c9ac29 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:72a9aa7b-2e8f-4905-8655-179fa9c9ac29', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:28:59,253 [236] [DEBUG] [app] Ending request: urn:request:2e5ed6b4-970c-4513-acad-4d3010ac2bc0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e5ed6b4-970c-4513-acad-4d3010ac2bc0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:28:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:28:59,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:28:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:28:59,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:28:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:28:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
exportactionlogsworker stdout | 2025-02-07 05:28:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:28:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:04.936688+00:00 (in 4.998334 seconds)
exportactionlogsworker stdout | 2025-02-07 05:28:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:59 UTC)" (scheduled at 2025-02-07 05:28:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:28:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:28:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:29:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:29:01,951 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:29:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:29:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:23.420886+00:00 (in 20.001298 seconds)
repositorygcworker stdout | 2025-02-07 05:29:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:03 UTC)" (scheduled at 2025-02-07 05:29:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:29:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:29:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 3, 419857), True, datetime.datetime(2025, 2, 7, 5, 29, 3, 419857), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:29:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:29:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:29:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:29:03,588 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:29:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:29:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:33.919286+00:00 (in 29.999553 seconds)
buildlogsarchiver stdout | 2025-02-07 05:29:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:33 UTC)" (scheduled at 2025-02-07 05:29:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:29:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 29, 3, 919920), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:29:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:29:03,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:29:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:29:04,788 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:29:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:29:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:52.937648+00:00 (in 48.000498 seconds)
exportactionlogsworker stdout | 2025-02-07 05:29:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:04 UTC)" (scheduled at 2025-02-07 05:29:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:29:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:29:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 4, 937458), True, datetime.datetime(2025, 2, 7, 5, 29, 4, 937458), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:29:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:29:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:29:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:29:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:29:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:18.732147+00:00 (in 9.999614 seconds)
notificationworker stdout | 2025-02-07 05:29:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:18 UTC)" (scheduled at 2025-02-07 05:29:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:29:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:29:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 8, 732803), True, datetime.datetime(2025, 2, 7, 5, 29, 8, 732803), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:29:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:29:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:29:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:29:09,134 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:29:10,087 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:29:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:29:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:24.423055+00:00 (in 13.998355 seconds)
gcworker stdout | 2025-02-07 05:29:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:40 UTC)" (scheduled at 2025-02-07 05:29:10.424250+00:00)
gcworker stdout | 2025-02-07 05:29:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:29:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905850431, None, 1, 0])
gcworker stdout | 2025-02-07 05:29:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:29:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:29:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:29:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:41.831521+00:00 (in 29.999504 seconds)
autopruneworker stdout | 2025-02-07 05:29:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:41 UTC)" (scheduled at 2025-02-07 05:29:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:29:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902551836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:29:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:29:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:29:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:29:14,055 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:29:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:29:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:29.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 05:29:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:29 UTC)" (scheduled at 2025-02-07 05:29:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:29:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:29:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:29:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:29:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:29:14,235 [235] [DEBUG] [app] Starting request: urn:request:eb746089-bff2-4b7e-81e1-3f91f06337bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:14,235 [238] [DEBUG] [app] Starting request: urn:request:d805b3f4-067f-4f91-889d-cc705acc7d66 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:29:14,239 [239] [DEBUG] [app] Starting request: urn:request:e5e04ea5-dbca-47b0-8d46-9fea11323616 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:14,239 [246] [DEBUG] [app] Starting request: urn:request:c08f9134-305b-4a1f-8071-10ec5a930367 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:14,239 [246] [DEBUG] [app] Ending request: urn:request:c08f9134-305b-4a1f-8071-10ec5a930367 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c08f9134-305b-4a1f-8071-10ec5a930367', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:29:14,240 [239] [DEBUG] [app] Ending request: urn:request:e5e04ea5-dbca-47b0-8d46-9fea11323616 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5e04ea5-dbca-47b0-8d46-9fea11323616', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:29:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:29:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:29:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:14,243 [236] [DEBUG] [app] Starting request: urn:request:0791a2bf-8052-4d3a-941d-875dcac2b99a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:29:14,243 [237] [DEBUG] [app] Starting request: urn:request:2b6e4738-fe59-4375-8cae-00079b10ed81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:29:14,243 [237] [DEBUG] [app] Ending request: urn:request:2b6e4738-fe59-4375-8cae-00079b10ed81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2b6e4738-fe59-4375-8cae-00079b10ed81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:29:14,243 [236] [DEBUG] [app] Ending request: urn:request:0791a2bf-8052-4d3a-941d-875dcac2b99a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0791a2bf-8052-4d3a-941d-875dcac2b99a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:29:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:29:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:29:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:29:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:14,254 [235] [DEBUG] [app] Ending request: urn:request:eb746089-bff2-4b7e-81e1-3f91f06337bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb746089-bff2-4b7e-81e1-3f91f06337bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:29:14,255 [238] [DEBUG] [app] Ending request: urn:request:d805b3f4-067f-4f91-889d-cc705acc7d66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d805b3f4-067f-4f91-889d-cc705acc7d66', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:29:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:29:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
builder stdout | 2025-02-07 05:29:14,645 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:29:15,236 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:29:16,632 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:29:17,402 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:29:17,460 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:29:17,752 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:29:17,755 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:29:17,760 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:29:17,762 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:29:17,765 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
autopruneworker stdout | 2025-02-07 05:29:18,718 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
notificationworker stdout | 2025-02-07 05:29:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:29:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:28.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 05:29:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:28 UTC)" (scheduled at 2025-02-07 05:29:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:29:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:29:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 18, 732852), True, datetime.datetime(2025, 2, 7, 5, 29, 18, 732852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:29:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:29:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:29:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:29:18,769 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:29:18,772 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:29:18,774 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:19,079 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:29:19,437 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:29:19,517 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:29:19,811 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:29:20,171 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:29:20,189 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:29:20,191 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:29:20,194 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:29:20,200 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:29:20,203 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:29:20,206 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:29:20,209 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:29:20,214 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:29:20,217 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gcworker stdout | 2025-02-07 05:29:20,501 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:29:20,813 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:29:20,910 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:29:22,476 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:29:22,525 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:29:22,973 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:29:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:29:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:03.419110+00:00 (in 39.997754 seconds)
repositorygcworker stdout | 2025-02-07 05:29:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:23 UTC)" (scheduled at 2025-02-07 05:29:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:29:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:29:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:29:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:29:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:40.424250+00:00 (in 16.000730 seconds)
gcworker stdout | 2025-02-07 05:29:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:54 UTC)" (scheduled at 2025-02-07 05:29:24.423055+00:00)
gcworker stdout | 2025-02-07 05:29:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:29:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:29:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:29:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:29:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:38.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 05:29:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:38 UTC)" (scheduled at 2025-02-07 05:29:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:29:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:29:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 28, 732845), True, datetime.datetime(2025, 2, 7, 5, 29, 28, 732845), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:29:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:29:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:29:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:29:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:29:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:44.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 05:29:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:44 UTC)" (scheduled at 2025-02-07 05:29:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:29:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:29:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:29:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:29:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:29:29,235 [235] [DEBUG] [app] Starting request: urn:request:f6d0be93-1757-489f-a848-9ccdb304bfd6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:29,235 [238] [DEBUG] [app] Starting request: urn:request:cea68d8f-d1e7-460b-87bb-2b55584dbf20 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:29:29,238 [243] [DEBUG] [app] Starting request: urn:request:af0b053f-74e6-46f5-b501-0053070ec909 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:29,238 [246] [DEBUG] [app] Starting request: urn:request:17bd6a8b-74f2-4372-8c3f-d040d9cca6bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:29,238 [243] [DEBUG] [app] Ending request: urn:request:af0b053f-74e6-46f5-b501-0053070ec909 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af0b053f-74e6-46f5-b501-0053070ec909', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:29:29,239 [246] [DEBUG] [app] Ending request: urn:request:17bd6a8b-74f2-4372-8c3f-d040d9cca6bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:17bd6a8b-74f2-4372-8c3f-d040d9cca6bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:29:29,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:29:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:29:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:29,241 [238] [DEBUG] [app] Starting request: urn:request:a0c37d15-df22-41d4-9149-70350b4422f1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:29:29,241 [238] [DEBUG] [app] Ending request: urn:request:a0c37d15-df22-41d4-9149-70350b4422f1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a0c37d15-df22-41d4-9149-70350b4422f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:29:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:29:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:29,242 [237] [DEBUG] [app] Starting request: urn:request:1e53e3ae-a93d-4c08-ab54-4e0a8c4393c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:29:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:29,242 [237] [DEBUG] [app] Ending request: urn:request:1e53e3ae-a93d-4c08-ab54-4e0a8c4393c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e53e3ae-a93d-4c08-ab54-4e0a8c4393c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:29:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:29:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:29,253 [238] [DEBUG] [app] Ending request: urn:request:cea68d8f-d1e7-460b-87bb-2b55584dbf20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cea68d8f-d1e7-460b-87bb-2b55584dbf20', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:29:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 05:29:29,254 [235] [DEBUG] [app] Ending request: urn:request:f6d0be93-1757-489f-a848-9ccdb304bfd6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f6d0be93-1757-489f-a848-9ccdb304bfd6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:29:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
namespacegcworker stdout | 2025-02-07 05:29:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:29:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:32.625945+00:00 (in 1.004361 seconds)
namespacegcworker stdout | 2025-02-07 05:29:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:31 UTC)" (scheduled at 2025-02-07 05:29:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:29:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:29:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 31, 621894), True, datetime.datetime(2025, 2, 7, 5, 29, 31, 621894), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:29:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:29:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:29:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:29:31,977 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:29:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:29:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:31.621131+00:00 (in 58.994720 seconds)
namespacegcworker stdout | 2025-02-07 05:29:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:32 UTC)" (scheduled at 2025-02-07 05:29:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:29:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:29:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:29:33,624 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:29:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:29:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:03.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 05:29:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:03 UTC)" (scheduled at 2025-02-07 05:29:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:29:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 29, 33, 919920), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:29:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:29:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:29:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:29:34,824 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:56.713547+00:00 (in 18.997188 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:37 UTC)" (scheduled at 2025-02-07 05:29:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:29:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:29:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:48.732147+00:00 (in 9.999601 seconds)
notificationworker stdout | 2025-02-07 05:29:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:48 UTC)" (scheduled at 2025-02-07 05:29:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:29:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:29:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 38, 732805), True, datetime.datetime(2025, 2, 7, 5, 29, 38, 732805), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:29:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:29:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:29:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:29:39,149 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:29:40,126 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:29:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:29:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:54.423055+00:00 (in 13.998345 seconds)
gcworker stdout | 2025-02-07 05:29:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:10 UTC)" (scheduled at 2025-02-07 05:29:40.424250+00:00)
gcworker stdout | 2025-02-07 05:29:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:29:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905880431, None, 1, 0])
gcworker stdout | 2025-02-07 05:29:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:29:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:29:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:29:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:11.831521+00:00 (in 29.999518 seconds)
autopruneworker stdout | 2025-02-07 05:29:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:11 UTC)" (scheduled at 2025-02-07 05:29:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:29:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902581836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:29:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:29:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:29:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:29:44,079 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:29:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:29:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:59.225378+00:00 (in 14.999545 seconds)
quotatotalworker stdout | 2025-02-07 05:29:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:59 UTC)" (scheduled at 2025-02-07 05:29:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:29:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:29:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:29:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:29:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:29:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:29:44,235 [236] [DEBUG] [app] Starting request: urn:request:4f4e7176-fa7a-44f0-94db-d72c0193fc33 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:44,235 [235] [DEBUG] [app] Starting request: urn:request:fff5f09b-eefb-40d7-8494-2a0cf0873be2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:29:44,238 [239] [DEBUG] [app] Starting request: urn:request:8d004556-60ca-4968-8c38-9437c11f53cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:44,238 [246] [DEBUG] [app] Starting request: urn:request:545c7caf-e802-4aa8-af61-ac5dcb1e47fd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:44,238 [239] [DEBUG] [app] Ending request: urn:request:8d004556-60ca-4968-8c38-9437c11f53cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d004556-60ca-4968-8c38-9437c11f53cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:29:44,238 [246] [DEBUG] [app] Ending request: urn:request:545c7caf-e802-4aa8-af61-ac5dcb1e47fd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:545c7caf-e802-4aa8-af61-ac5dcb1e47fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:29:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:29:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:29:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:44,242 [237] [DEBUG] [app] Starting request: urn:request:152dfa39-5a25-400d-8a5e-5e6f8deba434 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:29:44,242 [238] [DEBUG] [app] Starting request: urn:request:83a2a929-67fc-4730-b068-623b6a424c60 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:29:44,242 [237] [DEBUG] [app] Ending request: urn:request:152dfa39-5a25-400d-8a5e-5e6f8deba434 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:152dfa39-5a25-400d-8a5e-5e6f8deba434', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:29:44,242 [238] [DEBUG] [app] Ending request: urn:request:83a2a929-67fc-4730-b068-623b6a424c60 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:83a2a929-67fc-4730-b068-623b6a424c60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:29:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:29:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:29:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:44,254 [235] [DEBUG] [app] Ending request: urn:request:fff5f09b-eefb-40d7-8494-2a0cf0873be2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fff5f09b-eefb-40d7-8494-2a0cf0873be2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:29:44,255 [236] [DEBUG] [app] Ending request: urn:request:4f4e7176-fa7a-44f0-94db-d72c0193fc33 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f4e7176-fa7a-44f0-94db-d72c0193fc33', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:29:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:29:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
builder stdout | 2025-02-07 05:29:44,682 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:29:45,272 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:29:46,669 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:29:47,434 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:29:47,477 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:29:47,760 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:29:47,763 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:29:47,766 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:29:47,770 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:29:47,773 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:29:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:29:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:50.814677+00:00 (in 2.082122 seconds)
notificationworker stdout | 2025-02-07 05:29:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:58 UTC)" (scheduled at 2025-02-07 05:29:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:29:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:29:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 48, 732842), True, datetime.datetime(2025, 2, 7, 5, 29, 48, 732842), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:29:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:29:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:29:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:29:58 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:29:48,748 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:29:48,775 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:29:48,778 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:29:48,782 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:49,110 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:29:49,473 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:29:49,554 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:29:49,831 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:29:50,196 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:29:50,202 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:29:50,206 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
repositorygcworker stdout | 2025-02-07 05:29:50,209 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gunicorn-registry stdout | 2025-02-07 05:29:50,212 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:29:50,215 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:29:50,219 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:29:50,221 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:29:50,224 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:29:50,226 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gcworker stdout | 2025-02-07 05:29:50,537 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 05:29:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:29:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:58.732147+00:00 (in 7.917024 seconds)
notificationworker stdout | 2025-02-07 05:29:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:50 UTC)" (scheduled at 2025-02-07 05:29:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:29:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:29:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:50 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 05:29:50,850 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:29:50,931 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:29:52,512 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:29:52,560 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
exportactionlogsworker stdout | 2025-02-07 05:29:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:29:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:29:59.937899+00:00 (in 6.999801 seconds)
exportactionlogsworker stdout | 2025-02-07 05:29:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:34:52 UTC)" (scheduled at 2025-02-07 05:29:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:29:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 29, 52, 938369), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:29:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 29, 52, 938369), True, datetime.datetime(2025, 2, 7, 5, 29, 52, 938369), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:29:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 29, 52, 938369), True, datetime.datetime(2025, 2, 7, 5, 29, 52, 938369), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 29, 52, 938369), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:29:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:29:52,949 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:34:52 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:29:52,980 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:29:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:29:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:10.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 05:29:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:24 UTC)" (scheduled at 2025-02-07 05:29:54.423055+00:00)
gcworker stdout | 2025-02-07 05:29:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:29:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:29:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:29:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:56.118855+00:00 (in 59.999529 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:29:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:56 UTC)" (scheduled at 2025-02-07 05:29:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:29:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:29:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:29:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:37.715878+00:00 (in 41.001887 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:56 UTC)" (scheduled at 2025-02-07 05:29:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:29:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:29:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:29:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:08.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 05:29:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:08 UTC)" (scheduled at 2025-02-07 05:29:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:29:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:29:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 29, 58, 732833), True, datetime.datetime(2025, 2, 7, 5, 29, 58, 732833), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:29:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:29:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:29:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:29:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:29:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:14.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 05:29:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:14 UTC)" (scheduled at 2025-02-07 05:29:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:29:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:29:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:29:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:29:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:29:59,235 [237] [DEBUG] [app] Starting request: urn:request:f7f6824c-96b0-4519-a55c-699f4af1af40 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:59,235 [238] [DEBUG] [app] Starting request: urn:request:dcf93972-db66-41ea-8bf3-57f8f4fa3b5a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:29:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:29:59,238 [239] [DEBUG] [app] Starting request: urn:request:609151e6-6394-455a-a1ae-1509293d628b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:59,238 [244] [DEBUG] [app] Starting request: urn:request:b85c47a4-5d19-4cd4-a457-afa1c62856ab (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:29:59,238 [239] [DEBUG] [app] Ending request: urn:request:609151e6-6394-455a-a1ae-1509293d628b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:609151e6-6394-455a-a1ae-1509293d628b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:29:59,238 [244] [DEBUG] [app] Ending request: urn:request:b85c47a4-5d19-4cd4-a457-afa1c62856ab (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b85c47a4-5d19-4cd4-a457-afa1c62856ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:29:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:29:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:29:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:29:59,242 [236] [DEBUG] [app] Starting request: urn:request:227cf042-2418-4247-b2de-53a5b83551b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:29:59,242 [236] [DEBUG] [app] Ending request: urn:request:227cf042-2418-4247-b2de-53a5b83551b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:227cf042-2418-4247-b2de-53a5b83551b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:29:59,242 [235] [DEBUG] [app] Starting request: urn:request:a18bad27-77b6-48c9-8ee0-6503b927720a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:29:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:29:59,242 [235] [DEBUG] [app] Ending request: urn:request:a18bad27-77b6-48c9-8ee0-6503b927720a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a18bad27-77b6-48c9-8ee0-6503b927720a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:29:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:29:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:29:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:29:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:29:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:29:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:29:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:29:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:29:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:29:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:29:59,253 [238] [DEBUG] [app] Ending request: urn:request:dcf93972-db66-41ea-8bf3-57f8f4fa3b5a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcf93972-db66-41ea-8bf3-57f8f4fa3b5a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:29:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:29:59,254 [237] [DEBUG] [app] Ending request: urn:request:f7f6824c-96b0-4519-a55c-699f4af1af40 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f7f6824c-96b0-4519-a55c-699f4af1af40', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:29:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:29:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:29:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:29:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:29:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:29:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:04.936688+00:00 (in 4.998345 seconds)
exportactionlogsworker stdout | 2025-02-07 05:29:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:59 UTC)" (scheduled at 2025-02-07 05:29:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:29:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:29:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:30:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:30:02,013 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:30:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:30:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:23.420886+00:00 (in 20.001310 seconds)
repositorygcworker stdout | 2025-02-07 05:30:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:03 UTC)" (scheduled at 2025-02-07 05:30:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:30:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:30:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 3, 419873), True, datetime.datetime(2025, 2, 7, 5, 30, 3, 419873), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:30:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:30:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:30:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:30:03,649 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:30:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:30:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:33.919286+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2025-02-07 05:30:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:33 UTC)" (scheduled at 2025-02-07 05:30:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:30:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 30, 3, 920007), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:30:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:30:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:30:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:30:04,853 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:30:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:30:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:59.937899+00:00 (in 55.000786 seconds)
exportactionlogsworker stdout | 2025-02-07 05:30:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:04 UTC)" (scheduled at 2025-02-07 05:30:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:30:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:30:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 4, 937434), True, datetime.datetime(2025, 2, 7, 5, 30, 4, 937434), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:30:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:30:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:30:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:04 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:30:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:30:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:18.732147+00:00 (in 9.999572 seconds)
notificationworker stdout | 2025-02-07 05:30:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:18 UTC)" (scheduled at 2025-02-07 05:30:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:30:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:30:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 8, 732845), True, datetime.datetime(2025, 2, 7, 5, 30, 8, 732845), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:30:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:30:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:30:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:30:09,161 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:30:10,140 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:30:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:30:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:24.423055+00:00 (in 13.998378 seconds)
gcworker stdout | 2025-02-07 05:30:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:40 UTC)" (scheduled at 2025-02-07 05:30:10.424250+00:00)
gcworker stdout | 2025-02-07 05:30:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:30:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905910431, None, 1, 0])
gcworker stdout | 2025-02-07 05:30:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:30:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:30:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:30:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:41.831521+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2025-02-07 05:30:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:41 UTC)" (scheduled at 2025-02-07 05:30:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:30:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902611836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:30:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:30:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:30:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:30:14,116 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:30:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:30:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:29.225378+00:00 (in 14.999622 seconds)
quotatotalworker stdout | 2025-02-07 05:30:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:29 UTC)" (scheduled at 2025-02-07 05:30:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:30:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:30:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:30:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:30:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:30:14,235 [235] [DEBUG] [app] Starting request: urn:request:948c8d42-457d-4a07-821c-2c6f6d76d8ff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:14,235 [236] [DEBUG] [app] Starting request: urn:request:61c67825-b214-4f13-a446-f32cb27e85a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:30:14,239 [246] [DEBUG] [app] Starting request: urn:request:970fc136-f047-422a-a471-ccfbb50e7de2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:14,239 [245] [DEBUG] [app] Starting request: urn:request:19d537f6-ec2c-402e-9625-09e7b697b291 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:14,239 [246] [DEBUG] [app] Ending request: urn:request:970fc136-f047-422a-a471-ccfbb50e7de2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:970fc136-f047-422a-a471-ccfbb50e7de2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:30:14,239 [245] [DEBUG] [app] Ending request: urn:request:19d537f6-ec2c-402e-9625-09e7b697b291 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:19d537f6-ec2c-402e-9625-09e7b697b291', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:30:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:30:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:30:14,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:14,242 [236] [DEBUG] [app] Starting request: urn:request:19a3ffb3-65bc-4126-9082-7db306b64711 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:14,242 [236] [DEBUG] [app] Ending request: urn:request:19a3ffb3-65bc-4126-9082-7db306b64711 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:19a3ffb3-65bc-4126-9082-7db306b64711', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:30:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:30:14,243 [237] [DEBUG] [app] Starting request: urn:request:306d0016-32d5-489e-b9f3-0da475c004f2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:14,243 [237] [DEBUG] [app] Ending request: urn:request:306d0016-32d5-489e-b9f3-0da475c004f2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:306d0016-32d5-489e-b9f3-0da475c004f2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:30:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:30:14,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:30:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:30:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:14,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:14,254 [236] [DEBUG] [app] Ending request: urn:request:61c67825-b214-4f13-a446-f32cb27e85a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61c67825-b214-4f13-a446-f32cb27e85a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:30:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:30:14,255 [235] [DEBUG] [app] Ending request: urn:request:948c8d42-457d-4a07-821c-2c6f6d76d8ff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:948c8d42-457d-4a07-821c-2c6f6d76d8ff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:30:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
builder stdout | 2025-02-07 05:30:14,707 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:30:15,308 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:30:16,705 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:30:17,445 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:30:17,513 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:30:17,769 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:30:17,772 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:30:17,775 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:30:17,778 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:30:17,780 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:30:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:30:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:28.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:30:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:28 UTC)" (scheduled at 2025-02-07 05:30:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:30:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:30:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 18, 732881), True, datetime.datetime(2025, 2, 7, 5, 30, 18, 732881), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:30:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:30:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:30:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:28 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:30:18,780 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
gunicorn-secscan stdout | 2025-02-07 05:30:18,784 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:30:18,787 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:30:18,789 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:19,129 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:30:19,505 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:30:19,581 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:30:19,867 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:30:20,207 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:30:20,213 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:30:20,215 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:30:20,220 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:30:20,223 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:30:20,227 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:30:20,230 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:30:20,233 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:30:20,236 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:30:20,243 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:30:20,574 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:30:20,887 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:30:20,947 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:30:22,548 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:30:22,591 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:30:23,000 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:30:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:30:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:03.419110+00:00 (in 39.997749 seconds)
repositorygcworker stdout | 2025-02-07 05:30:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:23 UTC)" (scheduled at 2025-02-07 05:30:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:30:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:30:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:30:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:30:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:40.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 05:30:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:54 UTC)" (scheduled at 2025-02-07 05:30:24.423055+00:00)
gcworker stdout | 2025-02-07 05:30:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:30:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:30:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:30:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:30:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:38.732147+00:00 (in 9.999528 seconds)
notificationworker stdout | 2025-02-07 05:30:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:38 UTC)" (scheduled at 2025-02-07 05:30:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:30:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:30:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 28, 732967), True, datetime.datetime(2025, 2, 7, 5, 30, 28, 732967), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:30:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:30:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:30:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:30:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:30:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:44.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 05:30:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:44 UTC)" (scheduled at 2025-02-07 05:30:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:30:29,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:30:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:30:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:30:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:30:29,236 [235] [DEBUG] [app] Starting request: urn:request:7061c633-04ad-4910-a94b-c22fed6bae64 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:29,236 [238] [DEBUG] [app] Starting request: urn:request:26a63566-4f63-432a-b9b4-b2cb0961fad1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:30:29,238 [246] [DEBUG] [app] Starting request: urn:request:e21b11ec-135b-4e8d-92ea-2e36b29ee4e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:29,238 [239] [DEBUG] [app] Starting request: urn:request:eeea10e3-0b58-4734-91e8-06008d5307f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:29,239 [239] [DEBUG] [app] Ending request: urn:request:eeea10e3-0b58-4734-91e8-06008d5307f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eeea10e3-0b58-4734-91e8-06008d5307f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:30:29,239 [246] [DEBUG] [app] Ending request: urn:request:e21b11ec-135b-4e8d-92ea-2e36b29ee4e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e21b11ec-135b-4e8d-92ea-2e36b29ee4e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:30:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:30:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:30:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:29,241 [238] [DEBUG] [app] Starting request: urn:request:f8ffc3bb-a30b-4e6a-afd5-12d738db245c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:29,242 [237] [DEBUG] [app] Starting request: urn:request:11d539be-234f-458d-b47d-4418292f743a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:29,242 [238] [DEBUG] [app] Ending request: urn:request:f8ffc3bb-a30b-4e6a-afd5-12d738db245c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f8ffc3bb-a30b-4e6a-afd5-12d738db245c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:30:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:30:29,242 [237] [DEBUG] [app] Ending request: urn:request:11d539be-234f-458d-b47d-4418292f743a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:11d539be-234f-458d-b47d-4418292f743a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:30:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:30:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:30:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:30:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:30:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:29,253 [238] [DEBUG] [app] Ending request: urn:request:26a63566-4f63-432a-b9b4-b2cb0961fad1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26a63566-4f63-432a-b9b4-b2cb0961fad1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:29,253 [235] [DEBUG] [app] Ending request: urn:request:7061c633-04ad-4910-a94b-c22fed6bae64 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7061c633-04ad-4910-a94b-c22fed6bae64', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:30:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:30:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:30:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 05:30:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:30:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:32.625945+00:00 (in 1.004393 seconds)
namespacegcworker stdout | 2025-02-07 05:30:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:31 UTC)" (scheduled at 2025-02-07 05:30:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:30:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:30:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 31, 621826), True, datetime.datetime(2025, 2, 7, 5, 30, 31, 621826), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:30:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:30:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:30:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:30:32,050 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:30:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:30:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:31.621131+00:00 (in 58.994676 seconds)
namespacegcworker stdout | 2025-02-07 05:30:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:32 UTC)" (scheduled at 2025-02-07 05:30:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:30:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:30:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:30:33,686 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:30:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:30:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:03.919286+00:00 (in 29.999556 seconds)
buildlogsarchiver stdout | 2025-02-07 05:30:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:03 UTC)" (scheduled at 2025-02-07 05:30:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:30:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 30, 33, 919902), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:30:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:30:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:30:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:30:34,869 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:56.713547+00:00 (in 18.997204 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:37 UTC)" (scheduled at 2025-02-07 05:30:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:30:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:30:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:48.732147+00:00 (in 9.999486 seconds)
notificationworker stdout | 2025-02-07 05:30:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:48 UTC)" (scheduled at 2025-02-07 05:30:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:30:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:30:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 38, 732974), True, datetime.datetime(2025, 2, 7, 5, 30, 38, 732974), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:30:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:30:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:30:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:30:39,198 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:30:40,178 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:30:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:30:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:54.423055+00:00 (in 13.998337 seconds)
gcworker stdout | 2025-02-07 05:30:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:10 UTC)" (scheduled at 2025-02-07 05:30:40.424250+00:00)
gcworker stdout | 2025-02-07 05:30:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:30:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905940431, None, 1, 0])
gcworker stdout | 2025-02-07 05:30:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:30:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:30:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:30:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:11.831521+00:00 (in 29.999438 seconds)
autopruneworker stdout | 2025-02-07 05:30:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:11 UTC)" (scheduled at 2025-02-07 05:30:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:30:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902641836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:30:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:30:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:30:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:30:44,152 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:30:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:30:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:59.225378+00:00 (in 14.999613 seconds)
quotatotalworker stdout | 2025-02-07 05:30:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:59 UTC)" (scheduled at 2025-02-07 05:30:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:30:44,225 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:30:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:30:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:30:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:30:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:30:44,236 [238] [DEBUG] [app] Starting request: urn:request:85cab4af-fd60-4110-b848-ded98133fac3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:44,236 [235] [DEBUG] [app] Starting request: urn:request:eaefd858-8e94-45fd-bac6-8a535a0e1240 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:30:44,239 [245] [DEBUG] [app] Starting request: urn:request:fec39f0d-83c9-4345-9ca7-4357877ceecc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:44,239 [239] [DEBUG] [app] Starting request: urn:request:f984e82b-b733-4872-a3c2-66f60ae668e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:44,239 [245] [DEBUG] [app] Ending request: urn:request:fec39f0d-83c9-4345-9ca7-4357877ceecc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fec39f0d-83c9-4345-9ca7-4357877ceecc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:30:44,239 [239] [DEBUG] [app] Ending request: urn:request:f984e82b-b733-4872-a3c2-66f60ae668e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f984e82b-b733-4872-a3c2-66f60ae668e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:30:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:30:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:30:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:44,243 [236] [DEBUG] [app] Starting request: urn:request:a59c58ef-665b-4db6-a08a-e205a5104d47 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:44,243 [237] [DEBUG] [app] Starting request: urn:request:b2110338-3eed-48e4-9fc2-98175f346673 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:44,243 [236] [DEBUG] [app] Ending request: urn:request:a59c58ef-665b-4db6-a08a-e205a5104d47 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a59c58ef-665b-4db6-a08a-e205a5104d47', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:30:44,243 [237] [DEBUG] [app] Ending request: urn:request:b2110338-3eed-48e4-9fc2-98175f346673 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b2110338-3eed-48e4-9fc2-98175f346673', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:30:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:30:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:30:44,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:30:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:30:44,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:44,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:44,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:44,256 [238] [DEBUG] [app] Ending request: urn:request:85cab4af-fd60-4110-b848-ded98133fac3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:85cab4af-fd60-4110-b848-ded98133fac3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:44,256 [235] [DEBUG] [app] Ending request: urn:request:eaefd858-8e94-45fd-bac6-8a535a0e1240 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eaefd858-8e94-45fd-bac6-8a535a0e1240', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:30:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:30:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:30:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
builder stdout | 2025-02-07 05:30:44,733 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:30:45,317 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:30:46,742 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:30:47,481 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:30:47,549 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:30:47,778 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:30:47,781 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:30:47,784 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:30:47,787 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:30:47,789 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:30:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:30:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:50.814677+00:00 (in 2.082054 seconds)
notificationworker stdout | 2025-02-07 05:30:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:58 UTC)" (scheduled at 2025-02-07 05:30:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:30:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:30:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 48, 732901), True, datetime.datetime(2025, 2, 7, 5, 30, 48, 732901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:30:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:30:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:30:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:30:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:30:48,793 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:30:48,796 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:30:48,798 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
autopruneworker stdout | 2025-02-07 05:30:48,816 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:49,154 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:30:49,521 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:30:49,588 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:30:49,901 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:30:50,218 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:30:50,222 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:30:50,224 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:30:50,228 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:30:50,231 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:30:50,234 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:30:50,238 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:30:50,242 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:30:50,245 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:30:50,279 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:30:50,583 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 05:30:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:30:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:30:58.732147+00:00 (in 7.917033 seconds)
notificationworker stdout | 2025-02-07 05:30:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:50 UTC)" (scheduled at 2025-02-07 05:30:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:30:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:30:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:50 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 05:30:50,918 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:30:50,983 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:30:52,584 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:30:52,628 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:30:53,018 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:30:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:30:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:10.424250+00:00 (in 16.000681 seconds)
gcworker stdout | 2025-02-07 05:30:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:24 UTC)" (scheduled at 2025-02-07 05:30:54.423055+00:00)
gcworker stdout | 2025-02-07 05:30:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:30:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:30:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:30:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:56.118855+00:00 (in 59.999496 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:30:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:56 UTC)" (scheduled at 2025-02-07 05:30:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:30:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:30:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:30:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:37.715878+00:00 (in 41.001868 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:56 UTC)" (scheduled at 2025-02-07 05:30:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:30:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:30:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:30:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:08.732147+00:00 (in 9.999579 seconds)
notificationworker stdout | 2025-02-07 05:30:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:08 UTC)" (scheduled at 2025-02-07 05:30:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:30:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:30:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 30, 58, 732893), True, datetime.datetime(2025, 2, 7, 5, 30, 58, 732893), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:30:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:30:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:30:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:30:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:30:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:14.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 05:30:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:14 UTC)" (scheduled at 2025-02-07 05:30:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:30:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:30:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:30:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:30:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:30:59,235 [237] [DEBUG] [app] Starting request: urn:request:81b4ffdf-2e52-481f-be00-914d5c516791 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:59,235 [235] [DEBUG] [app] Starting request: urn:request:1e9ec2c5-c2e1-426d-96c1-d223e1e7f561 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:30:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:30:59,238 [244] [DEBUG] [app] Starting request: urn:request:37d43a4a-5e42-40a3-8eca-857f1dacd861 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:59,238 [239] [DEBUG] [app] Starting request: urn:request:321e3db1-cbb9-4407-953f-7ecfdeceb010 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:30:59,239 [244] [DEBUG] [app] Ending request: urn:request:37d43a4a-5e42-40a3-8eca-857f1dacd861 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:37d43a4a-5e42-40a3-8eca-857f1dacd861', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:30:59,239 [239] [DEBUG] [app] Ending request: urn:request:321e3db1-cbb9-4407-953f-7ecfdeceb010 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:321e3db1-cbb9-4407-953f-7ecfdeceb010', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:30:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:30:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:30:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:30:59,242 [235] [DEBUG] [app] Starting request: urn:request:5f0e8f1c-e746-420a-bdca-273c71924986 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:59,242 [235] [DEBUG] [app] Ending request: urn:request:5f0e8f1c-e746-420a-bdca-273c71924986 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5f0e8f1c-e746-420a-bdca-273c71924986', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:30:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:30:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:30:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:59,243 [238] [DEBUG] [app] Starting request: urn:request:1eb668f6-9e17-4c6a-9320-b14b6cc683c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:30:59,243 [238] [DEBUG] [app] Ending request: urn:request:1eb668f6-9e17-4c6a-9320-b14b6cc683c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1eb668f6-9e17-4c6a-9320-b14b6cc683c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:30:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:30:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:30:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:30:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:30:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:30:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:30:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:30:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:30:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:30:59,253 [235] [DEBUG] [app] Ending request: urn:request:1e9ec2c5-c2e1-426d-96c1-d223e1e7f561 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e9ec2c5-c2e1-426d-96c1-d223e1e7f561', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 05:30:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:30:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:30:59,255 [237] [DEBUG] [app] Ending request: urn:request:81b4ffdf-2e52-481f-be00-914d5c516791 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81b4ffdf-2e52-481f-be00-914d5c516791', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:30:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:30:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:30:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:30:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:30:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:30:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:04.936688+00:00 (in 4.998338 seconds)
exportactionlogsworker stdout | 2025-02-07 05:30:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:59 UTC)" (scheduled at 2025-02-07 05:30:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:30:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:30:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:31:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:31:02,068 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:31:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:31:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:23.420886+00:00 (in 20.001303 seconds)
repositorygcworker stdout | 2025-02-07 05:31:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:03 UTC)" (scheduled at 2025-02-07 05:31:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:31:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:31:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 3, 419863), True, datetime.datetime(2025, 2, 7, 5, 31, 3, 419863), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:31:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:31:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:31:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:31:03,706 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:31:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:31:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:33.919286+00:00 (in 29.999570 seconds)
buildlogsarchiver stdout | 2025-02-07 05:31:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:33 UTC)" (scheduled at 2025-02-07 05:31:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:31:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 31, 3, 919897), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:31:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:31:03,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:31:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:33 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:31:04,906 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2025-02-07 05:31:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:31:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:59.937899+00:00 (in 55.000789 seconds)
exportactionlogsworker stdout | 2025-02-07 05:31:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:04 UTC)" (scheduled at 2025-02-07 05:31:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:31:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:31:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 4, 937446), True, datetime.datetime(2025, 2, 7, 5, 31, 4, 937446), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:31:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:31:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:31:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:04 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,058 [55] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,058 [55] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 06:31:05.057762+00:00 (in 3599.999492 seconds)
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,058 [55] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-02-07 06:31:05 UTC)" (scheduled at 2025-02-07 05:31:05.057762+00:00)
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,058 [55] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,058 [55] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ...
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,062 [55] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,062 [55] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,062 [55] [DEBUG] [__main__] Performing blob upload cleanup
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,063 [55] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 5, 5, 31, 5, 62658), 1, 0])
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [__main__] No additional stale blob uploads found
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [data.database] Disconnecting from database.
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7fda72c4b8b0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7fda72c615e0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function remove_bucket_from_url_paths_from_model at 0x7fda72c4f430>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7fda72c61430>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x7fda72c4f5e0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.hooks] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.regions] Calling endpoint provider with parameters: {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3.openshift-storage.svc.cluster.local:443', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.regions] Endpoint provider result: https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.regions] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,069 [55] [DEBUG] [botocore.regions] Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7fda72c61940>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function add_recursion_detection_header at 0x7fda72c610d0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7fda72c4bca0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.endpoint] Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '?list-type=2', 'query_string': {'prefix': 'datastorage/registry/uploads', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61'}, 'body': b'', 'auth_path': '/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360/', 'url': 'https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fda6c1ad490>, 'has_streaming_input': False, 'auth_type': 'v4', 'encoding_type_auto_set': True, 's3_redirect': {'redirected': False, 'bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'params': {'Bucket': 'quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360', 'Prefix': 'datastorage/registry/uploads', 'EncodingType': 'url'}}, 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}}}
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fda6c1ad3a0>>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.ListObjectsV2: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fda6ddd5b80>>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7fda72c61310>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event before-sign.s3.ListObjectsV2: calling handler <function remove_arn_from_signing_path at 0x7fda72c4f550>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.auth] Calculating signature using v4 auth.
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.auth] CanonicalRequest:
blobuploadcleanupworker stdout | GET
blobuploadcleanupworker stdout | /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360
blobuploadcleanupworker stdout | encoding-type=url&list-type=2&prefix=datastorage%2Fregistry%2Fuploads
blobuploadcleanupworker stdout | host:s3.openshift-storage.svc.cluster.local
blobuploadcleanupworker stdout | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | x-amz-date:20250207T053105Z
blobuploadcleanupworker stdout | host;x-amz-content-sha256;x-amz-date
blobuploadcleanupworker stdout | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.auth] StringToSign:
blobuploadcleanupworker stdout | AWS4-HMAC-SHA256
blobuploadcleanupworker stdout | 20250207T053105Z
blobuploadcleanupworker stdout | 20250207/us-east-1/s3/aws4_request
blobuploadcleanupworker stdout | c2ad954f9de71be02bd34acc7c6cdf987507a0860d1d0c3d15df27d0375cba81
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.auth] Signature:
blobuploadcleanupworker stdout | 0c4b26d2333bd5fc226c0a30a34b2c3e7776e9030fab92b30f14573cc4ab1c06
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.hooks] Event request-created.s3.ListObjectsV2: calling handler <function add_retry_headers at 0x7fda72c4f3a0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.openshift-storage.svc.cluster.local:443/quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url, headers={'User-Agent': b'Boto3/1.28.61 md/Botocore#1.31.61 ua/2.0 os/linux#5.14.0-427.50.1.el9_4.x86_64 md/arch#x86_64 lang/python#3.9.20 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.31.61', 'X-Amz-Date': b'20250207T053105Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=pjqBQGfl18IdF9RWc0aJ/20250207/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0c4b26d2333bd5fc226c0a30a34b2c3e7776e9030fab92b30f14573cc4ab1c06', 'amz-sdk-invocation-id': b'd648560d-126f-49bb-ba49-a848a4807add', 'amz-sdk-request': b'attempt=1'}>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,070 [55] [DEBUG] [botocore.httpsession] Certificate path: /app/lib/python3.9/site-packages/certifi/cacert.pem
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,071 [55] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: s3.openshift-storage.svc.cluster.local
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,107 [55] [DEBUG] [urllib3.connectionpool] https://s3.openshift-storage.svc.cluster.local:443 "GET /quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360?list-type=2&prefix=datastorage%2Fregistry%2Fuploads&encoding-type=url HTTP/1.1" 200 411
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [botocore.parsers] Response headers: {'Server': 'NooBaa/5.17.3-427bfd2', 'x-amz-request-id': 'm6ubw5c2-9qafuq-5ng', 'x-amz-id-2': 'm6ubw5c2-9qafuq-5ng', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Access-Control-Allow-Methods': 'GET,POST,PUT,DELETE,OPTIONS', 'Access-Control-Allow-Headers': 'Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256,amz-sdk-invocation-id,amz-sdk-request', 'Access-Control-Expose-Headers': 'ETag,X-Amz-Version-Id', 'Content-Type': 'application/xml', 'Content-Length': '411', 'Date': 'Fri, 07 Feb 2025 05:31:05 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5'}
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [botocore.parsers] Response body:
blobuploadcleanupworker stdout | b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>quay-datastore-19c28dae-ad79-42c6-9e00-303fd5ad5360</Name><Prefix>datastorage%2Fregistry%2Fuploads</Prefix><Delimiter>undefined</Delimiter><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><EncodingType>url</EncodingType><StartAfter>undefined</StartAfter><KeyCount>0</KeyCount></ListBucketResult>'
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler <botocore.retryhandler.RetryHandler object at 0x7fda5fb4eac0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [botocore.retryhandler] No retry needed.
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [botocore.hooks] Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x7fda5fb4e9a0>>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [botocore.hooks] Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7fda72c4b9d0>
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,108 [55] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,109 [55] [DEBUG] [util.locking] Released lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2025-02-07 05:31:05,109 [55] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2025-02-07 06:31:05 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:31:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:31:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:18.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 05:31:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:18 UTC)" (scheduled at 2025-02-07 05:31:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:31:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:31:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 8, 732895), True, datetime.datetime(2025, 2, 7, 5, 31, 8, 732895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:31:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:31:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:31:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:31:09,234 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:31:10,214 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:31:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:31:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:24.423055+00:00 (in 13.998291 seconds)
gcworker stdout | 2025-02-07 05:31:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:40 UTC)" (scheduled at 2025-02-07 05:31:10.424250+00:00)
gcworker stdout | 2025-02-07 05:31:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:31:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738905970431, None, 1, 0])
gcworker stdout | 2025-02-07 05:31:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:31:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:31:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:31:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:41.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 05:31:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:41 UTC)" (scheduled at 2025-02-07 05:31:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:31:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902671836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:31:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:31:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:31:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:31:14,159 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:31:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:31:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:29.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 05:31:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:29 UTC)" (scheduled at 2025-02-07 05:31:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:31:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:31:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:31:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:31:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:31:14,235 [235] [DEBUG] [app] Starting request: urn:request:1b80f537-90ea-42d6-9ab9-3c9145608e6d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:14,235 [238] [DEBUG] [app] Starting request: urn:request:e0ab29d4-0efc-44eb-abc8-a95a1ac91ff1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:31:14,239 [239] [DEBUG] [app] Starting request: urn:request:f2de8139-7620-440c-b1d4-889e27079790 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:14,239 [245] [DEBUG] [app] Starting request: urn:request:a856c1b1-19c1-4c4a-81c8-eb637b0a3209 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:14,239 [239] [DEBUG] [app] Ending request: urn:request:f2de8139-7620-440c-b1d4-889e27079790 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f2de8139-7620-440c-b1d4-889e27079790', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:31:14,239 [245] [DEBUG] [app] Ending request: urn:request:a856c1b1-19c1-4c4a-81c8-eb637b0a3209 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a856c1b1-19c1-4c4a-81c8-eb637b0a3209', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:31:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:31:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:31:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:14,242 [238] [DEBUG] [app] Starting request: urn:request:b5c8af06-7204-4f75-95cf-927cebfe1e85 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:14,242 [238] [DEBUG] [app] Ending request: urn:request:b5c8af06-7204-4f75-95cf-927cebfe1e85 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b5c8af06-7204-4f75-95cf-927cebfe1e85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:14,242 [235] [DEBUG] [app] Starting request: urn:request:bf9a056e-df1e-445f-9ddb-0fdf4a009712 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:14,243 [235] [DEBUG] [app] Ending request: urn:request:bf9a056e-df1e-445f-9ddb-0fdf4a009712 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf9a056e-df1e-445f-9ddb-0fdf4a009712', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:31:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:31:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:14,254 [235] [DEBUG] [app] Ending request: urn:request:1b80f537-90ea-42d6-9ab9-3c9145608e6d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1b80f537-90ea-42d6-9ab9-3c9145608e6d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:14,254 [238] [DEBUG] [app] Ending request: urn:request:e0ab29d4-0efc-44eb-abc8-a95a1ac91ff1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0ab29d4-0efc-44eb-abc8-a95a1ac91ff1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:31:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:31:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:31:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
builder stdout | 2025-02-07 05:31:14,744 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:31:15,338 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:31:16,754 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:31:17,505 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:31:17,585 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:31:17,785 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:31:17,789 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:31:17,791 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:31:17,795 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:31:17,797 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:31:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:31:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:28.732147+00:00 (in 9.999579 seconds)
notificationworker stdout | 2025-02-07 05:31:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:28 UTC)" (scheduled at 2025-02-07 05:31:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:31:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:31:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 18, 732849), True, datetime.datetime(2025, 2, 7, 5, 31, 18, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:31:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:31:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:31:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:31:18,801 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:31:18,804 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:31:18,806 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
autopruneworker stdout | 2025-02-07 05:31:18,853 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:19,161 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:31:19,557 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:31:19,617 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:31:19,937 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:31:20,230 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:31:20,232 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:31:20,236 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:31:20,239 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:31:20,241 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:31:20,244 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:31:20,247 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:31:20,249 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:31:20,254 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:31:20,305 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:31:20,619 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:31:20,954 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:31:21,011 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:31:22,618 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:31:22,664 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:31:23,049 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:31:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:31:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:03.419110+00:00 (in 39.997761 seconds)
repositorygcworker stdout | 2025-02-07 05:31:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:23 UTC)" (scheduled at 2025-02-07 05:31:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:31:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:31:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:31:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:31:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:40.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 05:31:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:54 UTC)" (scheduled at 2025-02-07 05:31:24.423055+00:00)
gcworker stdout | 2025-02-07 05:31:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:31:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:31:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:31:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:31:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:38.732147+00:00 (in 9.999510 seconds)
notificationworker stdout | 2025-02-07 05:31:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:38 UTC)" (scheduled at 2025-02-07 05:31:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:31:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:31:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 28, 732949), True, datetime.datetime(2025, 2, 7, 5, 31, 28, 732949), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:31:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:31:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:31:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:31:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:31:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:44.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 05:31:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:44 UTC)" (scheduled at 2025-02-07 05:31:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:31:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:31:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:31:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:31:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:31:29,235 [237] [DEBUG] [app] Starting request: urn:request:a7d6de0a-c01d-4326-a6d2-23d05905bc03 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:29,235 [235] [DEBUG] [app] Starting request: urn:request:1b5d0d94-0d34-4176-acb2-9d603048b4dd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:31:29,238 [239] [DEBUG] [app] Starting request: urn:request:d4e7c30c-842e-42f5-88b6-babaa38c3bda (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:29,238 [246] [DEBUG] [app] Starting request: urn:request:26f5dbae-c9eb-4ea2-a5fa-079b8dc9187a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:29,239 [239] [DEBUG] [app] Ending request: urn:request:d4e7c30c-842e-42f5-88b6-babaa38c3bda (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d4e7c30c-842e-42f5-88b6-babaa38c3bda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:31:29,239 [246] [DEBUG] [app] Ending request: urn:request:26f5dbae-c9eb-4ea2-a5fa-079b8dc9187a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:26f5dbae-c9eb-4ea2-a5fa-079b8dc9187a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:31:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:31:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:31:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:29,241 [237] [DEBUG] [app] Starting request: urn:request:580a6e1d-578f-458b-898d-e35be30ec6d2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:29,242 [237] [DEBUG] [app] Ending request: urn:request:580a6e1d-578f-458b-898d-e35be30ec6d2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:580a6e1d-578f-458b-898d-e35be30ec6d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:31:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:29,242 [238] [DEBUG] [app] Starting request: urn:request:c5b6f02a-fb1d-43aa-bb11-5ac8e28d96b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:29,243 [238] [DEBUG] [app] Ending request: urn:request:c5b6f02a-fb1d-43aa-bb11-5ac8e28d96b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5b6f02a-fb1d-43aa-bb11-5ac8e28d96b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:31:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:29,253 [237] [DEBUG] [app] Ending request: urn:request:a7d6de0a-c01d-4326-a6d2-23d05905bc03 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a7d6de0a-c01d-4326-a6d2-23d05905bc03', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:31:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 05:31:29,254 [235] [DEBUG] [app] Ending request: urn:request:1b5d0d94-0d34-4176-acb2-9d603048b4dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1b5d0d94-0d34-4176-acb2-9d603048b4dd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:31:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 05:31:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:31:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:32.625945+00:00 (in 1.004333 seconds)
namespacegcworker stdout | 2025-02-07 05:31:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:31 UTC)" (scheduled at 2025-02-07 05:31:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:31:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:31:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 31, 621884), True, datetime.datetime(2025, 2, 7, 5, 31, 31, 621884), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:31:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:31:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:31:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:31:32,104 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:31:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:31:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:13.624301+00:00 (in 40.997898 seconds)
namespacegcworker stdout | 2025-02-07 05:31:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:32 UTC)" (scheduled at 2025-02-07 05:31:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:31:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:31:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:31:33,743 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:31:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:31:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:03.919286+00:00 (in 29.999542 seconds)
buildlogsarchiver stdout | 2025-02-07 05:31:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:03 UTC)" (scheduled at 2025-02-07 05:31:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:31:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 31, 33, 919927), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:31:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:31:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:31:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:31:34,922 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:56.713547+00:00 (in 18.997189 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:37 UTC)" (scheduled at 2025-02-07 05:31:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:31:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:31:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:48.732147+00:00 (in 9.999513 seconds)
notificationworker stdout | 2025-02-07 05:31:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:48 UTC)" (scheduled at 2025-02-07 05:31:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:31:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:31:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 38, 732924), True, datetime.datetime(2025, 2, 7, 5, 31, 38, 732924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:31:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:31:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:31:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:31:39,271 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:31:40,249 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:31:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:31:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:54.423055+00:00 (in 13.998324 seconds)
gcworker stdout | 2025-02-07 05:31:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:10 UTC)" (scheduled at 2025-02-07 05:31:40.424250+00:00)
gcworker stdout | 2025-02-07 05:31:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:31:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906000431, None, 1, 0])
gcworker stdout | 2025-02-07 05:31:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:31:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:31:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:31:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:11.831521+00:00 (in 29.999564 seconds)
autopruneworker stdout | 2025-02-07 05:31:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:11 UTC)" (scheduled at 2025-02-07 05:31:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:31:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902701836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:31:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:31:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:31:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:31:44,195 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:31:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:31:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:59.225378+00:00 (in 14.999561 seconds)
quotatotalworker stdout | 2025-02-07 05:31:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:59 UTC)" (scheduled at 2025-02-07 05:31:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:31:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:31:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:31:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:31:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:31:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:31:44,235 [236] [DEBUG] [app] Starting request: urn:request:104613e8-6ea5-4ca4-925b-c39d35d50d3b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:44,235 [238] [DEBUG] [app] Starting request: urn:request:02ac3acb-9a45-4772-b776-cf2c63d2d410 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:31:44,238 [244] [DEBUG] [app] Starting request: urn:request:fe5f06b2-5b26-43be-809c-cbaf1f2ae4f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:44,238 [246] [DEBUG] [app] Starting request: urn:request:6ffd06d1-ea34-4beb-890a-8183958e89ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:44,239 [244] [DEBUG] [app] Ending request: urn:request:fe5f06b2-5b26-43be-809c-cbaf1f2ae4f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fe5f06b2-5b26-43be-809c-cbaf1f2ae4f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:31:44,239 [246] [DEBUG] [app] Ending request: urn:request:6ffd06d1-ea34-4beb-890a-8183958e89ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ffd06d1-ea34-4beb-890a-8183958e89ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:31:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:31:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:31:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:44,242 [235] [DEBUG] [app] Starting request: urn:request:1771f0ea-6d76-461e-ab77-96ce08206171 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:44,242 [236] [DEBUG] [app] Starting request: urn:request:79f39b3c-5afc-417d-94dc-07491d392096 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:44,242 [236] [DEBUG] [app] Ending request: urn:request:79f39b3c-5afc-417d-94dc-07491d392096 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79f39b3c-5afc-417d-94dc-07491d392096', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:44,242 [235] [DEBUG] [app] Ending request: urn:request:1771f0ea-6d76-461e-ab77-96ce08206171 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1771f0ea-6d76-461e-ab77-96ce08206171', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:31:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:31:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:31:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:44,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:44,255 [236] [DEBUG] [app] Ending request: urn:request:104613e8-6ea5-4ca4-925b-c39d35d50d3b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:104613e8-6ea5-4ca4-925b-c39d35d50d3b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:44,255 [238] [DEBUG] [app] Ending request: urn:request:02ac3acb-9a45-4772-b776-cf2c63d2d410 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:02ac3acb-9a45-4772-b776-cf2c63d2d410', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:31:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.022)
gunicorn-web stdout | 2025-02-07 05:31:44,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:31:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
builder stdout | 2025-02-07 05:31:44,772 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:31:45,374 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:31:46,790 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:31:47,519 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:31:47,618 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:31:47,793 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:31:47,796 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:31:47,799 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:31:47,802 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:31:47,805 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:31:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:31:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:50.814677+00:00 (in 2.082046 seconds)
notificationworker stdout | 2025-02-07 05:31:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:58 UTC)" (scheduled at 2025-02-07 05:31:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:31:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:31:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 48, 732929), True, datetime.datetime(2025, 2, 7, 5, 31, 48, 732929), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:31:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:31:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:31:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:31:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:31:48,809 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:31:48,811 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
gunicorn-secscan stdout | 2025-02-07 05:31:48,814 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
autopruneworker stdout | 2025-02-07 05:31:48,889 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:49,174 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:31:49,595 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:31:49,629 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:31:49,973 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:31:50,239 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:31:50,243 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:31:50,246 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:31:50,249 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:31:50,252 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:31:50,256 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:31:50,259 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:31:50,261 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:31:50,265 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:31:50,332 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:31:50,655 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 05:31:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:31:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:31:58.732147+00:00 (in 7.916945 seconds)
notificationworker stdout | 2025-02-07 05:31:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:50 UTC)" (scheduled at 2025-02-07 05:31:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:31:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:31:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:50 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 05:31:50,972 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:31:51,047 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:31:52,654 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:31:52,671 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:31:53,085 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:31:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:31:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:10.424250+00:00 (in 16.000681 seconds)
gcworker stdout | 2025-02-07 05:31:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:24 UTC)" (scheduled at 2025-02-07 05:31:54.423055+00:00)
gcworker stdout | 2025-02-07 05:31:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:31:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:31:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:31:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:56.118855+00:00 (in 59.999499 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:31:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:56 UTC)" (scheduled at 2025-02-07 05:31:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:31:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:31:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:31:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:37.715878+00:00 (in 41.001792 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:56 UTC)" (scheduled at 2025-02-07 05:31:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:31:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:31:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:31:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:08.732147+00:00 (in 9.999490 seconds)
notificationworker stdout | 2025-02-07 05:31:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:08 UTC)" (scheduled at 2025-02-07 05:31:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:31:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:31:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 31, 58, 732992), True, datetime.datetime(2025, 2, 7, 5, 31, 58, 732992), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:31:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:31:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:31:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:31:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:31:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:14.225378+00:00 (in 14.999572 seconds)
quotatotalworker stdout | 2025-02-07 05:31:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:14 UTC)" (scheduled at 2025-02-07 05:31:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:31:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:31:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:31:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:31:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:31:59,235 [235] [DEBUG] [app] Starting request: urn:request:ba3de436-a324-49be-96d0-0fbe33a045b0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:59,235 [237] [DEBUG] [app] Starting request: urn:request:ae8e79e7-fd8e-4ab2-af38-d30fa29dd794 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:31:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:31:59,238 [246] [DEBUG] [app] Starting request: urn:request:344fdf2d-33ca-4d34-b42d-58ca2f2b8fa6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:59,238 [239] [DEBUG] [app] Starting request: urn:request:52e1c982-c3f7-4811-9922-bb140fa8c265 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:31:59,239 [246] [DEBUG] [app] Ending request: urn:request:344fdf2d-33ca-4d34-b42d-58ca2f2b8fa6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:344fdf2d-33ca-4d34-b42d-58ca2f2b8fa6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:31:59,239 [239] [DEBUG] [app] Ending request: urn:request:52e1c982-c3f7-4811-9922-bb140fa8c265 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:52e1c982-c3f7-4811-9922-bb140fa8c265', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:31:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:31:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:31:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:31:59,241 [236] [DEBUG] [app] Starting request: urn:request:589d2cf9-a289-4ee6-9902-e346ed84565e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:59,242 [236] [DEBUG] [app] Ending request: urn:request:589d2cf9-a289-4ee6-9902-e346ed84565e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:589d2cf9-a289-4ee6-9902-e346ed84565e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:59,242 [237] [DEBUG] [app] Starting request: urn:request:a51d0a07-86d9-4c58-bb16-27a84f76c743 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:31:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:31:59,242 [237] [DEBUG] [app] Ending request: urn:request:a51d0a07-86d9-4c58-bb16-27a84f76c743 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a51d0a07-86d9-4c58-bb16-27a84f76c743', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:31:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:31:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:31:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:31:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:31:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:31:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:31:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:31:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:31:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:31:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:31:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:31:59,254 [237] [DEBUG] [app] Ending request: urn:request:ae8e79e7-fd8e-4ab2-af38-d30fa29dd794 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ae8e79e7-fd8e-4ab2-af38-d30fa29dd794', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:31:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:31:59,255 [235] [DEBUG] [app] Ending request: urn:request:ba3de436-a324-49be-96d0-0fbe33a045b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba3de436-a324-49be-96d0-0fbe33a045b0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:31:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:31:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:31:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:31:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:31:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:31:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:04.936688+00:00 (in 4.998307 seconds)
exportactionlogsworker stdout | 2025-02-07 05:31:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:59 UTC)" (scheduled at 2025-02-07 05:31:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:31:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:31:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:32:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:32:02,122 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:32:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:32:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:23.420886+00:00 (in 20.001286 seconds)
repositorygcworker stdout | 2025-02-07 05:32:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:03 UTC)" (scheduled at 2025-02-07 05:32:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:32:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:32:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 3, 419909), True, datetime.datetime(2025, 2, 7, 5, 32, 3, 419909), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:32:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:32:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:32:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:32:03,779 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:32:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:32:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:33.919286+00:00 (in 29.999527 seconds)
buildlogsarchiver stdout | 2025-02-07 05:32:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:33 UTC)" (scheduled at 2025-02-07 05:32:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:32:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 32, 3, 920013), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:32:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:32:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:32:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:32:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:32:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:59.937899+00:00 (in 55.000720 seconds)
exportactionlogsworker stdout | 2025-02-07 05:32:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:04 UTC)" (scheduled at 2025-02-07 05:32:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:32:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:32:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 4, 937510), True, datetime.datetime(2025, 2, 7, 5, 32, 4, 937510), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:32:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:32:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:32:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:32:04,958 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:32:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:18.732147+00:00 (in 9.999509 seconds)
notificationworker stdout | 2025-02-07 05:32:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:18 UTC)" (scheduled at 2025-02-07 05:32:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:32:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:32:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 8, 732944), True, datetime.datetime(2025, 2, 7, 5, 32, 8, 732944), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:32:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:32:08,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:32:08,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:32:09,307 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:32:10,278 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:32:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:32:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:24.423055+00:00 (in 13.998252 seconds)
gcworker stdout | 2025-02-07 05:32:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:40 UTC)" (scheduled at 2025-02-07 05:32:10.424250+00:00)
gcworker stdout | 2025-02-07 05:32:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:32:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906030431, None, 1, 0])
gcworker stdout | 2025-02-07 05:32:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:32:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:32:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:32:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:41.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 05:32:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:41 UTC)" (scheduled at 2025-02-07 05:32:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:32:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902731836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:32:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:32:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:32:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:41 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:32:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:32:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:31.621131+00:00 (in 17.996397 seconds)
namespacegcworker stdout | 2025-02-07 05:32:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:37:13 UTC)" (scheduled at 2025-02-07 05:32:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:32:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 32, 13, 624981), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:32:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 32, 13, 624981), True, datetime.datetime(2025, 2, 7, 5, 32, 13, 624981), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:32:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 32, 13, 624981), True, datetime.datetime(2025, 2, 7, 5, 32, 13, 624981), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 32, 13, 624981), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:32:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:32:13,634 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:37:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:32:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:32:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:29.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 05:32:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:29 UTC)" (scheduled at 2025-02-07 05:32:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:32:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:32:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:32:14,233 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
quotatotalworker stdout | 2025-02-07 05:32:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:32:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:32:14,235 [235] [DEBUG] [app] Starting request: urn:request:bbe818fe-28b0-4be7-8c6e-17aed0f6082a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:14,235 [238] [DEBUG] [app] Starting request: urn:request:1c03a463-e5cc-4f48-b808-a8bac229fd74 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:32:14,238 [245] [DEBUG] [app] Starting request: urn:request:c5089ea1-9531-4c5c-a298-432f7b4cc59c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:14,239 [245] [DEBUG] [app] Ending request: urn:request:c5089ea1-9531-4c5c-a298-432f7b4cc59c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5089ea1-9531-4c5c-a298-432f7b4cc59c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:32:14,239 [239] [DEBUG] [app] Starting request: urn:request:7f089eb8-d65e-48ca-8314-9a611b29ea99 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:32:14,239 [239] [DEBUG] [app] Ending request: urn:request:7f089eb8-d65e-48ca-8314-9a611b29ea99 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7f089eb8-d65e-48ca-8314-9a611b29ea99', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:32:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:32:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:14,242 [237] [DEBUG] [app] Starting request: urn:request:66e95f86-0420-4e03-90b6-b632449a9a40 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:14,242 [237] [DEBUG] [app] Ending request: urn:request:66e95f86-0420-4e03-90b6-b632449a9a40 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66e95f86-0420-4e03-90b6-b632449a9a40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:14,243 [238] [DEBUG] [app] Starting request: urn:request:017f5770-fad0-4a54-952f-3cd4bc763d83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:32:14,243 [238] [DEBUG] [app] Ending request: urn:request:017f5770-fad0-4a54-952f-3cd4bc763d83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:017f5770-fad0-4a54-952f-3cd4bc763d83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:32:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:14,254 [238] [DEBUG] [app] Ending request: urn:request:1c03a463-e5cc-4f48-b808-a8bac229fd74 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c03a463-e5cc-4f48-b808-a8bac229fd74', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:14,254 [235] [DEBUG] [app] Ending request: urn:request:bbe818fe-28b0-4be7-8c6e-17aed0f6082a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bbe818fe-28b0-4be7-8c6e-17aed0f6082a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:32:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
builder stdout | 2025-02-07 05:32:14,802 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:32:15,401 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:32:16,819 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:32:17,533 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:32:17,654 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:32:17,803 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:32:17,806 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:32:17,809 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:32:17,812 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:32:17,815 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:32:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:28.732147+00:00 (in 9.999509 seconds)
notificationworker stdout | 2025-02-07 05:32:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:28 UTC)" (scheduled at 2025-02-07 05:32:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:32:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:32:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 18, 732944), True, datetime.datetime(2025, 2, 7, 5, 32, 18, 732944), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:32:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:32:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:32:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:32:18,818 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:32:18,823 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:32:18,830 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:32:18,925 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:19,207 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:32:19,629 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:32:19,641 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:32:19,990 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:32:20,249 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:32:20,252 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:32:20,255 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:32:20,258 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:32:20,263 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:32:20,266 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:32:20,270 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:32:20,273 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:32:20,276 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:32:20,352 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:32:20,674 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:32:21,002 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:32:21,081 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:32:22,690 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:32:22,707 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:32:23,116 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:32:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:32:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:03.419110+00:00 (in 39.997705 seconds)
repositorygcworker stdout | 2025-02-07 05:32:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:23 UTC)" (scheduled at 2025-02-07 05:32:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:32:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:32:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:32:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:32:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:40.424250+00:00 (in 16.000712 seconds)
gcworker stdout | 2025-02-07 05:32:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:54 UTC)" (scheduled at 2025-02-07 05:32:24.423055+00:00)
gcworker stdout | 2025-02-07 05:32:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:32:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:32:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:32:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:33.814166+00:00 (in 5.081493 seconds)
notificationworker stdout | 2025-02-07 05:32:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:38 UTC)" (scheduled at 2025-02-07 05:32:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:32:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:32:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 28, 732981), True, datetime.datetime(2025, 2, 7, 5, 32, 28, 732981), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:32:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:32:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:32:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:32:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:32:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:44.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 05:32:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:44 UTC)" (scheduled at 2025-02-07 05:32:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:32:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:32:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:32:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:32:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:32:29,236 [238] [DEBUG] [app] Starting request: urn:request:f9ad57db-a779-44d8-881b-ea8e2d058e08 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:29,236 [236] [DEBUG] [app] Starting request: urn:request:5192294e-f141-455b-9766-624eab467403 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:32:29,239 [239] [DEBUG] [app] Starting request: urn:request:32e382f4-a462-4dda-ab07-ed1fc42a1e7c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:29,239 [245] [DEBUG] [app] Starting request: urn:request:f7ce240f-f35d-41fe-b414-60b3d21035b4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:29,239 [239] [DEBUG] [app] Ending request: urn:request:32e382f4-a462-4dda-ab07-ed1fc42a1e7c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:32e382f4-a462-4dda-ab07-ed1fc42a1e7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:32:29,239 [245] [DEBUG] [app] Ending request: urn:request:f7ce240f-f35d-41fe-b414-60b3d21035b4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f7ce240f-f35d-41fe-b414-60b3d21035b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:32:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:32:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:32:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:29,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:29,242 [238] [DEBUG] [app] Starting request: urn:request:bb80eeea-ba54-4587-a1ff-774a4f09def3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:29,243 [238] [DEBUG] [app] Ending request: urn:request:bb80eeea-ba54-4587-a1ff-774a4f09def3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb80eeea-ba54-4587-a1ff-774a4f09def3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:32:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:29,243 [235] [DEBUG] [app] Starting request: urn:request:4baf9f3c-2af2-42dc-82ec-9937808203df (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:29,244 [235] [DEBUG] [app] Ending request: urn:request:4baf9f3c-2af2-42dc-82ec-9937808203df (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4baf9f3c-2af2-42dc-82ec-9937808203df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:32:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:32:29,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:29,245 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:29,245 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:29,249 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:29,253 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:29,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:29,254 [238] [DEBUG] [app] Ending request: urn:request:f9ad57db-a779-44d8-881b-ea8e2d058e08 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f9ad57db-a779-44d8-881b-ea8e2d058e08', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:32:29,255 [236] [DEBUG] [app] Ending request: urn:request:5192294e-f141-455b-9766-624eab467403 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5192294e-f141-455b-9766-624eab467403', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:29,256 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:29,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 05:32:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:32:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:32.625945+00:00 (in 1.004356 seconds)
namespacegcworker stdout | 2025-02-07 05:32:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:31 UTC)" (scheduled at 2025-02-07 05:32:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:32:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:32:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 31, 621864), True, datetime.datetime(2025, 2, 7, 5, 32, 31, 621864), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:32:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:32:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:32:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:32:32,150 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:32:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:32:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:31.621131+00:00 (in 58.994697 seconds)
namespacegcworker stdout | 2025-02-07 05:32:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:32 UTC)" (scheduled at 2025-02-07 05:32:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:32:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:32:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:32:33,786 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
notificationworker stdout | 2025-02-07 05:32:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:38.732147+00:00 (in 4.917452 seconds)
notificationworker stdout | 2025-02-07 05:32:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:37:33 UTC)" (scheduled at 2025-02-07 05:32:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:32:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 32, 33, 814988), 'notification/%'])
notificationworker stdout | 2025-02-07 05:32:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 32, 33, 814988), True, datetime.datetime(2025, 2, 7, 5, 32, 33, 814988), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:32:33,824 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 32, 33, 814988), True, datetime.datetime(2025, 2, 7, 5, 32, 33, 814988), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 32, 33, 814988), 'notification/%'])
notificationworker stdout | 2025-02-07 05:32:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:32:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:37:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:32:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:32:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:03.919286+00:00 (in 29.999480 seconds)
buildlogsarchiver stdout | 2025-02-07 05:32:33,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:03 UTC)" (scheduled at 2025-02-07 05:32:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:32:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 32, 33, 920159), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:32:33,928 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:32:33,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:32:33,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:32:34,994 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:56.713547+00:00 (in 18.997139 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:37 UTC)" (scheduled at 2025-02-07 05:32:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:32:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:48.732147+00:00 (in 9.999518 seconds)
notificationworker stdout | 2025-02-07 05:32:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:48 UTC)" (scheduled at 2025-02-07 05:32:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:32:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:32:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 38, 732893), True, datetime.datetime(2025, 2, 7, 5, 32, 38, 732893), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:32:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:32:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:32:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:32:39,315 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:32:40,292 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:32:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:32:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:54.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 05:32:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:10 UTC)" (scheduled at 2025-02-07 05:32:40.424250+00:00)
gcworker stdout | 2025-02-07 05:32:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:32:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906060430, None, 1, 0])
gcworker stdout | 2025-02-07 05:32:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:32:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:32:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:32:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:11.831521+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2025-02-07 05:32:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:11 UTC)" (scheduled at 2025-02-07 05:32:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:32:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902761836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:32:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:32:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:32:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:32:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:32:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:59.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 05:32:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:59 UTC)" (scheduled at 2025-02-07 05:32:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:32:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:32:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:32:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:32:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:32:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:32:44,235 [237] [DEBUG] [app] Starting request: urn:request:55d72508-a814-4e51-bcf2-cfb649513202 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:44,235 [235] [DEBUG] [app] Starting request: urn:request:ac3bd0cc-72f9-4830-bf33-5545cc0b50be (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:32:44,239 [246] [DEBUG] [app] Starting request: urn:request:cdaad5b9-92e0-4a05-b542-887b6382ac55 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:44,239 [239] [DEBUG] [app] Starting request: urn:request:d6a38f29-58a0-4d45-ba11-aa8f9fccc232 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:44,239 [246] [DEBUG] [app] Ending request: urn:request:cdaad5b9-92e0-4a05-b542-887b6382ac55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cdaad5b9-92e0-4a05-b542-887b6382ac55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:32:44,239 [239] [DEBUG] [app] Ending request: urn:request:d6a38f29-58a0-4d45-ba11-aa8f9fccc232 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6a38f29-58a0-4d45-ba11-aa8f9fccc232', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:32:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:32:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:32:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:44,242 [237] [DEBUG] [app] Starting request: urn:request:9343493b-6047-4b52-8d25-9cf73452d552 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:44,242 [238] [DEBUG] [app] Starting request: urn:request:b49811b3-ff09-499e-9f12-a0a3f79b45c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:44,242 [237] [DEBUG] [app] Ending request: urn:request:9343493b-6047-4b52-8d25-9cf73452d552 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9343493b-6047-4b52-8d25-9cf73452d552', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:44,242 [238] [DEBUG] [app] Ending request: urn:request:b49811b3-ff09-499e-9f12-a0a3f79b45c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b49811b3-ff09-499e-9f12-a0a3f79b45c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:32:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:32:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:44,253 [237] [DEBUG] [app] Ending request: urn:request:55d72508-a814-4e51-bcf2-cfb649513202 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55d72508-a814-4e51-bcf2-cfb649513202', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:44,253 [235] [DEBUG] [app] Ending request: urn:request:ac3bd0cc-72f9-4830-bf33-5545cc0b50be (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac3bd0cc-72f9-4830-bf33-5545cc0b50be', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:32:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:32:44,269 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:32:44,838 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:32:45,437 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:32:46,855 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:32:47,568 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:32:47,668 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:32:47,810 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:32:47,813 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:32:47,816 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:32:47,819 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:32:47,823 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:32:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:50.814677+00:00 (in 2.082094 seconds)
notificationworker stdout | 2025-02-07 05:32:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:58 UTC)" (scheduled at 2025-02-07 05:32:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:32:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:32:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 48, 732854), True, datetime.datetime(2025, 2, 7, 5, 32, 48, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:32:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:32:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:32:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:32:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:32:48,826 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:32:48,829 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:32:48,837 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:32:48,935 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:49,243 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:32:49,658 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:32:49,676 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:32:50,002 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:32:50,259 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:32:50,262 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:32:50,264 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:32:50,269 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:32:50,274 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:32:50,279 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:32:50,281 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:32:50,284 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:32:50,286 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:32:50,388 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:32:50,710 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 05:32:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:32:58.732147+00:00 (in 7.917019 seconds)
notificationworker stdout | 2025-02-07 05:32:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:50 UTC)" (scheduled at 2025-02-07 05:32:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:32:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:32:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:50 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 05:32:51,018 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:32:51,116 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:32:52,717 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:32:52,720 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:32:53,144 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:32:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:32:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:10.424250+00:00 (in 16.000755 seconds)
gcworker stdout | 2025-02-07 05:32:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:24 UTC)" (scheduled at 2025-02-07 05:32:54.423055+00:00)
gcworker stdout | 2025-02-07 05:32:54,423 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 05:32:54,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:32:54,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:32:54,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:32:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:32:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:56.118855+00:00 (in 59.999539 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:32:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:56 UTC)" (scheduled at 2025-02-07 05:32:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:32:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:32:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:32:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:37.715878+00:00 (in 41.001880 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:56 UTC)" (scheduled at 2025-02-07 05:32:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:32:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:32:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:32:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:08.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:32:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:08 UTC)" (scheduled at 2025-02-07 05:32:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:32:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:32:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 32, 58, 732865), True, datetime.datetime(2025, 2, 7, 5, 32, 58, 732865), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:32:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:32:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:32:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:32:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:32:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:14.225378+00:00 (in 14.999517 seconds)
quotatotalworker stdout | 2025-02-07 05:32:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:14 UTC)" (scheduled at 2025-02-07 05:32:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:32:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:32:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:32:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:32:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:32:59,235 [238] [DEBUG] [app] Starting request: urn:request:2d072f09-0e89-4645-87a2-ec3bfbd75356 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:59,235 [236] [DEBUG] [app] Starting request: urn:request:9cb013b7-0fe0-412a-91a3-b9f40f99a19d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:32:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:32:59,238 [239] [DEBUG] [app] Starting request: urn:request:f3d52007-fc69-46e1-a78d-59b9a2151d00 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:59,238 [245] [DEBUG] [app] Starting request: urn:request:783b7349-0c8a-47f7-8ddd-ecfe7d79db13 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:32:59,238 [239] [DEBUG] [app] Ending request: urn:request:f3d52007-fc69-46e1-a78d-59b9a2151d00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f3d52007-fc69-46e1-a78d-59b9a2151d00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:32:59,238 [245] [DEBUG] [app] Ending request: urn:request:783b7349-0c8a-47f7-8ddd-ecfe7d79db13 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:783b7349-0c8a-47f7-8ddd-ecfe7d79db13', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:32:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:32:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:32:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:32:59,241 [238] [DEBUG] [app] Starting request: urn:request:0a550b93-172b-487d-948c-8af715aab980 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:59,242 [237] [DEBUG] [app] Starting request: urn:request:873f393e-23d3-4a35-bc6d-8e6823837ba6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:32:59,242 [238] [DEBUG] [app] Ending request: urn:request:0a550b93-172b-487d-948c-8af715aab980 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a550b93-172b-487d-948c-8af715aab980', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:59,242 [237] [DEBUG] [app] Ending request: urn:request:873f393e-23d3-4a35-bc6d-8e6823837ba6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:873f393e-23d3-4a35-bc6d-8e6823837ba6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:32:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:32:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:32:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:32:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:32:59,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:32:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:32:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:32:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:32:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:32:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:32:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:32:59,254 [236] [DEBUG] [app] Ending request: urn:request:9cb013b7-0fe0-412a-91a3-b9f40f99a19d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9cb013b7-0fe0-412a-91a3-b9f40f99a19d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:59,254 [238] [DEBUG] [app] Ending request: urn:request:2d072f09-0e89-4645-87a2-ec3bfbd75356 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d072f09-0e89-4645-87a2-ec3bfbd75356', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:32:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:32:59,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:32:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:32:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:32:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 05:32:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:32:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:04.936688+00:00 (in 4.998279 seconds)
exportactionlogsworker stdout | 2025-02-07 05:32:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:59 UTC)" (scheduled at 2025-02-07 05:32:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:32:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:32:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:33:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:33:02,157 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:33:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:33:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:23.420886+00:00 (in 20.001316 seconds)
repositorygcworker stdout | 2025-02-07 05:33:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:03 UTC)" (scheduled at 2025-02-07 05:33:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:33:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:33:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 3, 419850), True, datetime.datetime(2025, 2, 7, 5, 33, 3, 419850), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:33:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:33:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:33:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:33:03,823 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:33:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:33:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:33.919286+00:00 (in 29.999496 seconds)
buildlogsarchiver stdout | 2025-02-07 05:33:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:33 UTC)" (scheduled at 2025-02-07 05:33:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:33:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 33, 3, 920071), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:33:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:33:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:33:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:33:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:33:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:59.937899+00:00 (in 55.000770 seconds)
exportactionlogsworker stdout | 2025-02-07 05:33:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:04 UTC)" (scheduled at 2025-02-07 05:33:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:33:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:33:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 4, 937433), True, datetime.datetime(2025, 2, 7, 5, 33, 4, 937433), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:33:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:33:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:33:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:33:05,004 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:33:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:33:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:18.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 05:33:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:18 UTC)" (scheduled at 2025-02-07 05:33:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:33:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:33:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 8, 732857), True, datetime.datetime(2025, 2, 7, 5, 33, 8, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:33:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:33:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:33:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:33:09,330 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:33:10,301 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:33:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:33:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:24.423055+00:00 (in 13.998368 seconds)
gcworker stdout | 2025-02-07 05:33:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:40 UTC)" (scheduled at 2025-02-07 05:33:10.424250+00:00)
gcworker stdout | 2025-02-07 05:33:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:33:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906090431, None, 1, 0])
gcworker stdout | 2025-02-07 05:33:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:33:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:33:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:33:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:41.831521+00:00 (in 29.999521 seconds)
autopruneworker stdout | 2025-02-07 05:33:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:41 UTC)" (scheduled at 2025-02-07 05:33:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:33:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902791836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:33:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:33:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:33:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:33:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:33:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:29.225378+00:00 (in 14.999519 seconds)
quotatotalworker stdout | 2025-02-07 05:33:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:29 UTC)" (scheduled at 2025-02-07 05:33:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:33:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:33:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:33:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:33:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:33:14,235 [235] [DEBUG] [app] Starting request: urn:request:42209652-4260-4a6c-ace0-38124226de68 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:14,235 [238] [DEBUG] [app] Starting request: urn:request:b5f624d4-e907-4ff0-b719-a1b83a4e4613 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:33:14,238 [245] [DEBUG] [app] Starting request: urn:request:56f1a90b-f232-4fca-bda2-b9347ade02d3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:14,238 [239] [DEBUG] [app] Starting request: urn:request:e2cba637-af86-40f5-9274-25e440fe7b69 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:14,238 [245] [DEBUG] [app] Ending request: urn:request:56f1a90b-f232-4fca-bda2-b9347ade02d3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56f1a90b-f232-4fca-bda2-b9347ade02d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:33:14,239 [239] [DEBUG] [app] Ending request: urn:request:e2cba637-af86-40f5-9274-25e440fe7b69 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e2cba637-af86-40f5-9274-25e440fe7b69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:33:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:33:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:14,241 [236] [DEBUG] [app] Starting request: urn:request:e4e82885-edbf-4013-ba47-524898de3e2e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:14,242 [235] [DEBUG] [app] Starting request: urn:request:7ccf0d1e-30a5-443d-8de5-11b04f2bbaee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:14,242 [236] [DEBUG] [app] Ending request: urn:request:e4e82885-edbf-4013-ba47-524898de3e2e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e4e82885-edbf-4013-ba47-524898de3e2e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:33:14,242 [235] [DEBUG] [app] Ending request: urn:request:7ccf0d1e-30a5-443d-8de5-11b04f2bbaee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7ccf0d1e-30a5-443d-8de5-11b04f2bbaee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:33:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:33:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:14,254 [235] [DEBUG] [app] Ending request: urn:request:42209652-4260-4a6c-ace0-38124226de68 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42209652-4260-4a6c-ace0-38124226de68', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:14,254 [238] [DEBUG] [app] Ending request: urn:request:b5f624d4-e907-4ff0-b719-a1b83a4e4613 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5f624d4-e907-4ff0-b719-a1b83a4e4613', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:33:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:33:14,285 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:33:14,857 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:33:15,473 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:33:16,891 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:33:17,593 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:33:17,705 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:33:17,819 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:33:17,823 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:33:17,827 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:33:17,830 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:33:17,833 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:33:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:33:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:28.732147+00:00 (in 9.999532 seconds)
notificationworker stdout | 2025-02-07 05:33:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:28 UTC)" (scheduled at 2025-02-07 05:33:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:33:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:33:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 18, 732904), True, datetime.datetime(2025, 2, 7, 5, 33, 18, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:33:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:33:18,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:33:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:33:18,836 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:33:18,839 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:33:18,844 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:33:18,972 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:19,258 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:33:19,694 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:33:19,709 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:33:20,038 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:33:20,267 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:33:20,273 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:33:20,275 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:33:20,278 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:33:20,286 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:33:20,288 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:33:20,291 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:33:20,295 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:33:20,297 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
repositorygcworker stdout | 2025-02-07 05:33:20,410 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:33:20,746 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:33:21,054 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:33:21,140 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:33:22,750 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:33:22,755 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:33:23,180 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:33:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:33:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:32.420707+00:00 (in 8.999373 seconds)
repositorygcworker stdout | 2025-02-07 05:33:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:23 UTC)" (scheduled at 2025-02-07 05:33:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:33:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:33:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:33:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:33:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:40.424250+00:00 (in 16.000746 seconds)
gcworker stdout | 2025-02-07 05:33:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:54 UTC)" (scheduled at 2025-02-07 05:33:24.423055+00:00)
gcworker stdout | 2025-02-07 05:33:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:33:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:33:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:33:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:33:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:38.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 05:33:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:38 UTC)" (scheduled at 2025-02-07 05:33:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:33:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:33:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 28, 732877), True, datetime.datetime(2025, 2, 7, 5, 33, 28, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:33:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:33:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:33:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:33:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:33:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:44.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 05:33:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:44 UTC)" (scheduled at 2025-02-07 05:33:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:33:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:33:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:33:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:33:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:33:29,235 [238] [DEBUG] [app] Starting request: urn:request:dd5e11af-34b1-4e60-b26e-93111f92be88 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:29,235 [235] [DEBUG] [app] Starting request: urn:request:1d9bb03f-ab3f-48b9-8eac-f92c7ebc47f7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:33:29,238 [246] [DEBUG] [app] Starting request: urn:request:e76c5102-e072-41d0-bfeb-cabe5c7c72ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:29,238 [241] [DEBUG] [app] Starting request: urn:request:2c92af2d-54b3-4fc9-b62d-76cc287af4b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:29,239 [241] [DEBUG] [app] Ending request: urn:request:2c92af2d-54b3-4fc9-b62d-76cc287af4b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c92af2d-54b3-4fc9-b62d-76cc287af4b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:33:29,239 [246] [DEBUG] [app] Ending request: urn:request:e76c5102-e072-41d0-bfeb-cabe5c7c72ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e76c5102-e072-41d0-bfeb-cabe5c7c72ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:33:29,239 [241] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:33:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:29,242 [235] [DEBUG] [app] Starting request: urn:request:a7c17e4f-a131-4b25-a858-0ce3c742276d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:29,242 [236] [DEBUG] [app] Starting request: urn:request:6f37c8c8-4953-43a7-93c7-ca90aab28dd0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:29,242 [235] [DEBUG] [app] Ending request: urn:request:a7c17e4f-a131-4b25-a858-0ce3c742276d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7c17e4f-a131-4b25-a858-0ce3c742276d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:33:29,242 [236] [DEBUG] [app] Ending request: urn:request:6f37c8c8-4953-43a7-93c7-ca90aab28dd0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6f37c8c8-4953-43a7-93c7-ca90aab28dd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:33:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:33:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:33:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:29,254 [235] [DEBUG] [app] Ending request: urn:request:1d9bb03f-ab3f-48b9-8eac-f92c7ebc47f7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1d9bb03f-ab3f-48b9-8eac-f92c7ebc47f7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:29,254 [238] [DEBUG] [app] Ending request: urn:request:dd5e11af-34b1-4e60-b26e-93111f92be88 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dd5e11af-34b1-4e60-b26e-93111f92be88', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:33:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 05:33:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:33:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:32.625945+00:00 (in 1.004358 seconds)
namespacegcworker stdout | 2025-02-07 05:33:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:31 UTC)" (scheduled at 2025-02-07 05:33:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:33:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:33:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 31, 621856), True, datetime.datetime(2025, 2, 7, 5, 33, 31, 621856), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:33:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:33:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:33:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:33:32,186 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:33:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:33:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:03.419110+00:00 (in 30.997964 seconds)
repositorygcworker stdout | 2025-02-07 05:33:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:38:32 UTC)" (scheduled at 2025-02-07 05:33:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:33:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 33, 32, 421419), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:33:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 33, 32, 421419), True, datetime.datetime(2025, 2, 7, 5, 33, 32, 421419), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:33:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 33, 32, 421419), True, datetime.datetime(2025, 2, 7, 5, 33, 32, 421419), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 33, 32, 421419), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:33:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:33:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:38:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:33:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:33:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:31.621131+00:00 (in 58.994718 seconds)
namespacegcworker stdout | 2025-02-07 05:33:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:32 UTC)" (scheduled at 2025-02-07 05:33:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:33:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:33:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:33:33,831 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:33:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:33:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:03.919286+00:00 (in 29.999530 seconds)
buildlogsarchiver stdout | 2025-02-07 05:33:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:03 UTC)" (scheduled at 2025-02-07 05:33:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:33:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 33, 33, 920007), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:33:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:33:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:33:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:33:35,027 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:56.713547+00:00 (in 18.997221 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:37 UTC)" (scheduled at 2025-02-07 05:33:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:33:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:33:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:48.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 05:33:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:48 UTC)" (scheduled at 2025-02-07 05:33:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:33:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:33:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 38, 732868), True, datetime.datetime(2025, 2, 7, 5, 33, 38, 732868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:33:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:33:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:33:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:33:39,343 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:33:40,318 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:33:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:33:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:54.423055+00:00 (in 13.998371 seconds)
gcworker stdout | 2025-02-07 05:33:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:10 UTC)" (scheduled at 2025-02-07 05:33:40.424250+00:00)
gcworker stdout | 2025-02-07 05:33:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:33:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906120431, None, 1, 0])
gcworker stdout | 2025-02-07 05:33:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:33:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:33:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:33:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:11.831521+00:00 (in 29.999486 seconds)
autopruneworker stdout | 2025-02-07 05:33:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:11 UTC)" (scheduled at 2025-02-07 05:33:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:33:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902821836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:33:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:33:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:33:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:33:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:33:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:59.225378+00:00 (in 14.999578 seconds)
quotatotalworker stdout | 2025-02-07 05:33:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:59 UTC)" (scheduled at 2025-02-07 05:33:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:33:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:33:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:33:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:33:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:33:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:33:44,235 [235] [DEBUG] [app] Starting request: urn:request:2a2b8a1f-b181-4eab-9806-badc54b9725f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:44,235 [237] [DEBUG] [app] Starting request: urn:request:1c45ce10-8651-49d0-9a52-036cb6116856 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:33:44,238 [239] [DEBUG] [app] Starting request: urn:request:1194d77c-cdda-4ca9-a0fd-2e810d926d9e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:44,238 [246] [DEBUG] [app] Starting request: urn:request:116a50ba-ca23-4467-acaa-88cad856dd19 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:44,238 [239] [DEBUG] [app] Ending request: urn:request:1194d77c-cdda-4ca9-a0fd-2e810d926d9e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1194d77c-cdda-4ca9-a0fd-2e810d926d9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:33:44,238 [246] [DEBUG] [app] Ending request: urn:request:116a50ba-ca23-4467-acaa-88cad856dd19 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:116a50ba-ca23-4467-acaa-88cad856dd19', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:33:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:33:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:33:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:44,241 [238] [DEBUG] [app] Starting request: urn:request:5bc2dc57-5750-4244-b289-e4fe2df0a7dd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:44,241 [238] [DEBUG] [app] Ending request: urn:request:5bc2dc57-5750-4244-b289-e4fe2df0a7dd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5bc2dc57-5750-4244-b289-e4fe2df0a7dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:33:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:44,242 [235] [DEBUG] [app] Starting request: urn:request:12a22ca2-3b0f-434a-abe0-22a5a62cb344 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:44,242 [235] [DEBUG] [app] Ending request: urn:request:12a22ca2-3b0f-434a-abe0-22a5a62cb344 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:12a22ca2-3b0f-434a-abe0-22a5a62cb344', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:33:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:33:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:44,253 [235] [DEBUG] [app] Ending request: urn:request:2a2b8a1f-b181-4eab-9806-badc54b9725f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a2b8a1f-b181-4eab-9806-badc54b9725f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:44,253 [237] [DEBUG] [app] Ending request: urn:request:1c45ce10-8651-49d0-9a52-036cb6116856 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c45ce10-8651-49d0-9a52-036cb6116856', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:33:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:33:44,321 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:33:44,893 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:33:45,505 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:33:46,922 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:33:47,629 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:33:47,730 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:33:47,827 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:33:47,831 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:33:47,835 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:33:47,837 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:33:47,840 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:33:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:33:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:50.814677+00:00 (in 2.082099 seconds)
notificationworker stdout | 2025-02-07 05:33:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:58 UTC)" (scheduled at 2025-02-07 05:33:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:33:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:33:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 48, 732885), True, datetime.datetime(2025, 2, 7, 5, 33, 48, 732885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:33:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:33:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:33:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:33:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:33:48,844 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:33:48,846 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:33:48,851 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:33:49,008 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:49,274 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:33:49,730 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:33:49,744 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:33:50,074 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:33:50,275 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:33:50,285 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:33:50,288 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:33:50,293 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:33:50,296 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:33:50,299 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:33:50,302 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:33:50,305 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:33:50,308 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
repositorygcworker stdout | 2025-02-07 05:33:50,435 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:33:50,783 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
notificationworker stdout | 2025-02-07 05:33:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:33:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:33:58.732147+00:00 (in 7.917012 seconds)
notificationworker stdout | 2025-02-07 05:33:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:50 UTC)" (scheduled at 2025-02-07 05:33:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:33:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:33:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:50 UTC)" executed successfully
securityscanningnotificationworker stdout | 2025-02-07 05:33:51,075 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:33:51,172 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:33:52,780 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:33:52,783 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:33:53,212 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:33:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:33:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:10.424250+00:00 (in 16.000738 seconds)
gcworker stdout | 2025-02-07 05:33:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:24 UTC)" (scheduled at 2025-02-07 05:33:54.423055+00:00)
gcworker stdout | 2025-02-07 05:33:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:33:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:33:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:33:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:56.118855+00:00 (in 59.999519 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:33:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:56 UTC)" (scheduled at 2025-02-07 05:33:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:33:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:33:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:33:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:37.715878+00:00 (in 41.001893 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:56 UTC)" (scheduled at 2025-02-07 05:33:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:33:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:33:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:33:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:08.732147+00:00 (in 9.999580 seconds)
notificationworker stdout | 2025-02-07 05:33:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:08 UTC)" (scheduled at 2025-02-07 05:33:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:33:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:33:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 33, 58, 732879), True, datetime.datetime(2025, 2, 7, 5, 33, 58, 732879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:33:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:33:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:33:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:33:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:33:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:14.225378+00:00 (in 14.999555 seconds)
quotatotalworker stdout | 2025-02-07 05:33:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:14 UTC)" (scheduled at 2025-02-07 05:33:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:33:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:33:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:33:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:33:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:33:59,235 [237] [DEBUG] [app] Starting request: urn:request:a93a5418-deff-4203-948a-ee6a161a54a9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:59,235 [235] [DEBUG] [app] Starting request: urn:request:3a0fc4bf-2fd3-4fb1-882e-eb98dc645de4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:33:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:33:59,239 [246] [DEBUG] [app] Starting request: urn:request:d619654c-83b3-415f-91e5-34810c45f2e6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:59,239 [239] [DEBUG] [app] Starting request: urn:request:facb9baf-1c9e-41cf-bcf9-bb62848f1d4e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:33:59,239 [239] [DEBUG] [app] Ending request: urn:request:facb9baf-1c9e-41cf-bcf9-bb62848f1d4e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:facb9baf-1c9e-41cf-bcf9-bb62848f1d4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:33:59,239 [246] [DEBUG] [app] Ending request: urn:request:d619654c-83b3-415f-91e5-34810c45f2e6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d619654c-83b3-415f-91e5-34810c45f2e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:33:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:33:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:33:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:33:59,242 [235] [DEBUG] [app] Starting request: urn:request:7208a290-e69f-4aed-a93f-ae9eb639e384 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:59,242 [237] [DEBUG] [app] Starting request: urn:request:ce56e208-fed5-4af3-807c-f7fb823d3836 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:33:59,242 [235] [DEBUG] [app] Ending request: urn:request:7208a290-e69f-4aed-a93f-ae9eb639e384 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7208a290-e69f-4aed-a93f-ae9eb639e384', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:33:59,242 [237] [DEBUG] [app] Ending request: urn:request:ce56e208-fed5-4af3-807c-f7fb823d3836 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ce56e208-fed5-4af3-807c-f7fb823d3836', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:33:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:33:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:33:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:33:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:33:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:33:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:33:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:33:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:33:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:33:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:33:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:33:59,254 [237] [DEBUG] [app] Ending request: urn:request:a93a5418-deff-4203-948a-ee6a161a54a9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a93a5418-deff-4203-948a-ee6a161a54a9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:59,254 [235] [DEBUG] [app] Ending request: urn:request:3a0fc4bf-2fd3-4fb1-882e-eb98dc645de4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a0fc4bf-2fd3-4fb1-882e-eb98dc645de4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:33:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:33:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:33:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:33:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:33:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:33:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:33:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:04.936688+00:00 (in 4.998330 seconds)
exportactionlogsworker stdout | 2025-02-07 05:33:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:59 UTC)" (scheduled at 2025-02-07 05:33:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:33:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:33:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:34:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:34:02,222 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:34:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:34:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:23.420886+00:00 (in 20.001306 seconds)
repositorygcworker stdout | 2025-02-07 05:34:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:03 UTC)" (scheduled at 2025-02-07 05:34:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:34:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:34:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 3, 419855), True, datetime.datetime(2025, 2, 7, 5, 34, 3, 419855), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:34:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:34:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:34:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:34:03,868 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:34:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:34:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:33.919286+00:00 (in 29.999559 seconds)
buildlogsarchiver stdout | 2025-02-07 05:34:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:33 UTC)" (scheduled at 2025-02-07 05:34:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:34:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 34, 3, 919910), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:34:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:34:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:34:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:34:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:34:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:52.937648+00:00 (in 48.000527 seconds)
exportactionlogsworker stdout | 2025-02-07 05:34:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:04 UTC)" (scheduled at 2025-02-07 05:34:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:34:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:34:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 4, 937470), True, datetime.datetime(2025, 2, 7, 5, 34, 4, 937470), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:34:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:34:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:34:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:34:05,040 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:34:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:34:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:18.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:34:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:18 UTC)" (scheduled at 2025-02-07 05:34:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:34:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:34:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 8, 732835), True, datetime.datetime(2025, 2, 7, 5, 34, 8, 732835), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:34:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:34:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:34:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:34:09,380 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:34:10,354 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:34:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:34:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:24.423055+00:00 (in 13.998361 seconds)
gcworker stdout | 2025-02-07 05:34:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:40 UTC)" (scheduled at 2025-02-07 05:34:10.424250+00:00)
gcworker stdout | 2025-02-07 05:34:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:34:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906150431, None, 1, 0])
gcworker stdout | 2025-02-07 05:34:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:34:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:34:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:34:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:41.831521+00:00 (in 29.999493 seconds)
autopruneworker stdout | 2025-02-07 05:34:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:41 UTC)" (scheduled at 2025-02-07 05:34:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:34:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902851836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:34:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:34:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:34:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:34:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:34:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:29.225378+00:00 (in 14.999571 seconds)
quotatotalworker stdout | 2025-02-07 05:34:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:29 UTC)" (scheduled at 2025-02-07 05:34:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:34:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:34:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:34:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:34:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:34:14,236 [237] [DEBUG] [app] Starting request: urn:request:777e3ee0-c3b9-4ddc-8ae4-2e04f96349df (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:14,236 [235] [DEBUG] [app] Starting request: urn:request:08d36b93-f8e1-4d0f-b603-97806e90843f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:34:14,239 [246] [DEBUG] [app] Starting request: urn:request:1c67f4ed-072e-4570-9f45-07538d9b6881 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:14,239 [245] [DEBUG] [app] Starting request: urn:request:43aac8fd-c32f-497a-b639-3c7854442b35 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:14,239 [246] [DEBUG] [app] Ending request: urn:request:1c67f4ed-072e-4570-9f45-07538d9b6881 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c67f4ed-072e-4570-9f45-07538d9b6881', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:34:14,239 [245] [DEBUG] [app] Ending request: urn:request:43aac8fd-c32f-497a-b639-3c7854442b35 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:43aac8fd-c32f-497a-b639-3c7854442b35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:34:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:34:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:14,242 [238] [DEBUG] [app] Starting request: urn:request:ae4d601b-dcba-4ebb-81f8-796cacca37b1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:34:14,242 [238] [DEBUG] [app] Ending request: urn:request:ae4d601b-dcba-4ebb-81f8-796cacca37b1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae4d601b-dcba-4ebb-81f8-796cacca37b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:34:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:14,242 [237] [DEBUG] [app] Starting request: urn:request:b97dc645-940d-4b95-bda3-ebbe67f218a9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:34:14,243 [237] [DEBUG] [app] Ending request: urn:request:b97dc645-940d-4b95-bda3-ebbe67f218a9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b97dc645-940d-4b95-bda3-ebbe67f218a9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:34:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:34:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:34:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:34:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:34:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:34:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:14,255 [237] [DEBUG] [app] Ending request: urn:request:777e3ee0-c3b9-4ddc-8ae4-2e04f96349df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:777e3ee0-c3b9-4ddc-8ae4-2e04f96349df', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:14,255 [235] [DEBUG] [app] Ending request: urn:request:08d36b93-f8e1-4d0f-b603-97806e90843f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08d36b93-f8e1-4d0f-b603-97806e90843f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:34:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:34:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:34:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:34:14,357 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:34:14,928 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:34:15,541 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:34:16,958 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:34:17,643 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:34:17,741 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:34:17,836 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:34:17,839 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:34:17,842 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:34:17,846 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:34:17,848 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:34:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:34:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:28.732147+00:00 (in 9.999531 seconds)
notificationworker stdout | 2025-02-07 05:34:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:28 UTC)" (scheduled at 2025-02-07 05:34:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:34:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:34:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 18, 732911), True, datetime.datetime(2025, 2, 7, 5, 34, 18, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:34:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:34:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:34:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:34:18,852 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:34:18,855 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:34:18,859 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:34:19,038 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:19,287 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:34:19,767 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:34:19,780 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:34:20,111 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:34:20,287 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:34:20,295 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:34:20,299 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:34:20,302 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:34:20,305 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:34:20,309 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:34:20,311 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:34:20,315 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:34:20,318 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
repositorygcworker stdout | 2025-02-07 05:34:20,470 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:34:20,799 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:34:21,112 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:34:21,198 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:34:22,814 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:34:22,818 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:34:23,248 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:34:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:34:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:03.419110+00:00 (in 39.997766 seconds)
repositorygcworker stdout | 2025-02-07 05:34:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:23 UTC)" (scheduled at 2025-02-07 05:34:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:34:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:34:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:34:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:34:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:40.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 05:34:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:54 UTC)" (scheduled at 2025-02-07 05:34:24.423055+00:00)
gcworker stdout | 2025-02-07 05:34:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:34:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:34:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:34:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:34:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:38.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:34:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:38 UTC)" (scheduled at 2025-02-07 05:34:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:34:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:34:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 28, 732861), True, datetime.datetime(2025, 2, 7, 5, 34, 28, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:34:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:34:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:34:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:34:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:34:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:44.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 05:34:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:44 UTC)" (scheduled at 2025-02-07 05:34:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:34:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:34:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:34:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:34:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:34:29,235 [235] [DEBUG] [app] Starting request: urn:request:c3bc7e3d-5a2a-4fb1-934f-17db00b40595 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:29,235 [238] [DEBUG] [app] Starting request: urn:request:598063b0-02f9-4f5c-bcf4-cd4fd6a71b36 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:34:29,238 [239] [DEBUG] [app] Starting request: urn:request:4964d90c-a182-40cd-844e-06fc50ffc1d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:29,238 [246] [DEBUG] [app] Starting request: urn:request:0f3633b2-96b1-4dca-af29-a55f892bbab2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:29,238 [239] [DEBUG] [app] Ending request: urn:request:4964d90c-a182-40cd-844e-06fc50ffc1d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4964d90c-a182-40cd-844e-06fc50ffc1d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:34:29,238 [246] [DEBUG] [app] Ending request: urn:request:0f3633b2-96b1-4dca-af29-a55f892bbab2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f3633b2-96b1-4dca-af29-a55f892bbab2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:34:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:34:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:29,241 [235] [DEBUG] [app] Starting request: urn:request:aef83d82-3433-4cf1-b048-b35d34ba432f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:34:29,241 [235] [DEBUG] [app] Ending request: urn:request:aef83d82-3433-4cf1-b048-b35d34ba432f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aef83d82-3433-4cf1-b048-b35d34ba432f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:34:29,242 [237] [DEBUG] [app] Starting request: urn:request:814abf77-e002-4676-9ffd-73c1101c82ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:34:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:29,242 [237] [DEBUG] [app] Ending request: urn:request:814abf77-e002-4676-9ffd-73c1101c82ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:814abf77-e002-4676-9ffd-73c1101c82ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:34:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:34:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:34:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:34:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:34:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:34:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:29,253 [235] [DEBUG] [app] Ending request: urn:request:c3bc7e3d-5a2a-4fb1-934f-17db00b40595 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3bc7e3d-5a2a-4fb1-934f-17db00b40595', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:29,253 [238] [DEBUG] [app] Ending request: urn:request:598063b0-02f9-4f5c-bcf4-cd4fd6a71b36 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:598063b0-02f9-4f5c-bcf4-cd4fd6a71b36', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:34:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:34:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:34:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:34:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:34:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:32.625945+00:00 (in 1.004339 seconds)
namespacegcworker stdout | 2025-02-07 05:34:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:31 UTC)" (scheduled at 2025-02-07 05:34:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:34:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:34:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 31, 621887), True, datetime.datetime(2025, 2, 7, 5, 34, 31, 621887), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:34:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:34:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:34:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:34:32,258 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:34:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:34:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:31.621131+00:00 (in 58.994735 seconds)
namespacegcworker stdout | 2025-02-07 05:34:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:32 UTC)" (scheduled at 2025-02-07 05:34:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:34:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:34:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:32 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:34:33,899 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
buildlogsarchiver stdout | 2025-02-07 05:34:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:34:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:03.919286+00:00 (in 29.999517 seconds)
buildlogsarchiver stdout | 2025-02-07 05:34:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:03 UTC)" (scheduled at 2025-02-07 05:34:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:34:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 34, 33, 919960), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:34:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:34:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:34:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:03 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:34:35,076 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:56.713547+00:00 (in 18.997209 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:37 UTC)" (scheduled at 2025-02-07 05:34:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:34:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:34:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:48.732147+00:00 (in 9.999573 seconds)
notificationworker stdout | 2025-02-07 05:34:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:48 UTC)" (scheduled at 2025-02-07 05:34:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:34:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:34:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 38, 732852), True, datetime.datetime(2025, 2, 7, 5, 34, 38, 732852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:34:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:34:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:34:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:34:39,417 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:34:40,390 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:34:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:34:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:54.423055+00:00 (in 13.998322 seconds)
gcworker stdout | 2025-02-07 05:34:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:10 UTC)" (scheduled at 2025-02-07 05:34:40.424250+00:00)
gcworker stdout | 2025-02-07 05:34:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:34:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906180431, None, 1, 0])
gcworker stdout | 2025-02-07 05:34:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:34:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:10 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:34:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:34:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:11.831521+00:00 (in 29.999518 seconds)
autopruneworker stdout | 2025-02-07 05:34:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:11 UTC)" (scheduled at 2025-02-07 05:34:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:34:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902881836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:34:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:34:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:34:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:34:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:34:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:59.225378+00:00 (in 14.999484 seconds)
quotatotalworker stdout | 2025-02-07 05:34:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:59 UTC)" (scheduled at 2025-02-07 05:34:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:34:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:34:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:34:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:34:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:34:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:34:44,235 [237] [DEBUG] [app] Starting request: urn:request:f7fad78a-4649-4282-b3f1-810c0713fc67 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:44,235 [238] [DEBUG] [app] Starting request: urn:request:42cef407-97d2-449d-ba56-a78dbfdf4228 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:34:44,239 [244] [DEBUG] [app] Starting request: urn:request:36956812-015a-4748-b992-4c18a8ae8fa9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:44,239 [242] [DEBUG] [app] Starting request: urn:request:4d150415-cb2f-47d6-92ba-b2bd6b33bbe9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:44,240 [242] [DEBUG] [app] Ending request: urn:request:4d150415-cb2f-47d6-92ba-b2bd6b33bbe9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d150415-cb2f-47d6-92ba-b2bd6b33bbe9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:34:44,240 [244] [DEBUG] [app] Ending request: urn:request:36956812-015a-4748-b992-4c18a8ae8fa9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36956812-015a-4748-b992-4c18a8ae8fa9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:34:44,240 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:34:44,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:34:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:44,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:44,243 [237] [DEBUG] [app] Starting request: urn:request:8c3c2d63-df98-47d7-8b55-ec8e4d844a34 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:34:44,243 [237] [DEBUG] [app] Ending request: urn:request:8c3c2d63-df98-47d7-8b55-ec8e4d844a34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8c3c2d63-df98-47d7-8b55-ec8e4d844a34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:34:44,244 [235] [DEBUG] [app] Starting request: urn:request:dc2b53c5-baa3-498b-9bc9-73b9b5304d3c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:34:44,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:44,244 [235] [DEBUG] [app] Ending request: urn:request:dc2b53c5-baa3-498b-9bc9-73b9b5304d3c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dc2b53c5-baa3-498b-9bc9-73b9b5304d3c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:34:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:44,245 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:44,245 [237] [DEBUG] [data.model.health] Validating database connection.
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:34:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:34:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:44,246 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:34:44,246 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:34:44,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:44,250 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:44,250 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:44,254 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:44,255 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:44,256 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:44,256 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:44,257 [237] [DEBUG] [app] Ending request: urn:request:f7fad78a-4649-4282-b3f1-810c0713fc67 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f7fad78a-4649-4282-b3f1-810c0713fc67', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:44,257 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:34:44,258 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.024 118 0.023)
gunicorn-web stdout | 2025-02-07 05:34:44,258 [238] [DEBUG] [app] Ending request: urn:request:42cef407-97d2-449d-ba56-a78dbfdf4228 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42cef407-97d2-449d-ba56-a78dbfdf4228', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:44,258 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:34:44,259 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.025 118 0.024)
quotatotalworker stdout | 2025-02-07 05:34:44,395 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:34:44,964 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:34:45,561 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:34:46,994 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:34:47,679 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:34:47,777 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:34:47,843 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:34:47,847 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:34:47,850 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:34:47,853 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:34:47,855 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:34:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:34:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:50.814677+00:00 (in 2.082106 seconds)
notificationworker stdout | 2025-02-07 05:34:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:58 UTC)" (scheduled at 2025-02-07 05:34:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:34:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:34:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 48, 732856), True, datetime.datetime(2025, 2, 7, 5, 34, 48, 732856), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:34:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:34:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:34:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:34:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:34:48,860 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:34:48,863 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:34:48,866 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:34:49,074 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:49,317 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:34:49,797 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:34:49,815 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:34:50,147 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:34:50,298 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:34:50,305 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:34:50,309 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:34:50,313 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:34:50,316 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:34:50,319 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:34:50,323 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:34:50,326 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:34:50,328 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:34:50,490 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:34:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:34:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:58.732147+00:00 (in 7.917004 seconds)
notificationworker stdout | 2025-02-07 05:34:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:50 UTC)" (scheduled at 2025-02-07 05:34:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:34:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:34:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:34:50,836 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:34:51,125 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:34:51,206 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:34:52,851 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:34:52,853 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
exportactionlogsworker stdout | 2025-02-07 05:34:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:34:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:34:59.937899+00:00 (in 6.999826 seconds)
exportactionlogsworker stdout | 2025-02-07 05:34:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:39:52 UTC)" (scheduled at 2025-02-07 05:34:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:34:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 34, 52, 938337), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:34:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 34, 52, 938337), True, datetime.datetime(2025, 2, 7, 5, 34, 52, 938337), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:34:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 34, 52, 938337), True, datetime.datetime(2025, 2, 7, 5, 34, 52, 938337), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 34, 52, 938337), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:34:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:34:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:39:52 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:34:53,270 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:34:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:34:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:10.424250+00:00 (in 16.000734 seconds)
gcworker stdout | 2025-02-07 05:34:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:24 UTC)" (scheduled at 2025-02-07 05:34:54.423055+00:00)
gcworker stdout | 2025-02-07 05:34:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:34:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:34:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:34:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:56.118855+00:00 (in 59.999521 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:34:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:56 UTC)" (scheduled at 2025-02-07 05:34:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:34:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:34:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:34:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:37.715878+00:00 (in 41.001874 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:56 UTC)" (scheduled at 2025-02-07 05:34:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:34:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:34:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:34:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:08.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 05:34:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:08 UTC)" (scheduled at 2025-02-07 05:34:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:34:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:34:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 34, 58, 732881), True, datetime.datetime(2025, 2, 7, 5, 34, 58, 732881), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:34:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:34:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:34:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:34:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:34:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:14.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 05:34:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:14 UTC)" (scheduled at 2025-02-07 05:34:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:34:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:34:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:34:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:34:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:34:59,235 [237] [DEBUG] [app] Starting request: urn:request:3a33d688-8309-4313-926b-b5660c541fa5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:59,235 [236] [DEBUG] [app] Starting request: urn:request:dc831fa7-4717-4a02-9070-709113ab622a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:34:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:34:59,238 [239] [DEBUG] [app] Starting request: urn:request:0c28cc65-2920-44b6-902b-c99341d44a5d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:59,238 [245] [DEBUG] [app] Starting request: urn:request:13ef9c37-177e-4893-907b-38b0d5798293 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:34:59,239 [239] [DEBUG] [app] Ending request: urn:request:0c28cc65-2920-44b6-902b-c99341d44a5d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c28cc65-2920-44b6-902b-c99341d44a5d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:34:59,239 [245] [DEBUG] [app] Ending request: urn:request:13ef9c37-177e-4893-907b-38b0d5798293 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13ef9c37-177e-4893-907b-38b0d5798293', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:34:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:34:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:34:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:34:59,242 [237] [DEBUG] [app] Starting request: urn:request:a6454a78-6511-46ae-8ff0-bb071e992272 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:34:59,242 [237] [DEBUG] [app] Ending request: urn:request:a6454a78-6511-46ae-8ff0-bb071e992272 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6454a78-6511-46ae-8ff0-bb071e992272', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:34:59,242 [235] [DEBUG] [app] Starting request: urn:request:78ab0fd7-6b48-45af-a7df-a215ea085c6f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:34:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:34:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:59,242 [235] [DEBUG] [app] Ending request: urn:request:78ab0fd7-6b48-45af-a7df-a215ea085c6f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78ab0fd7-6b48-45af-a7df-a215ea085c6f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:34:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:34:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:34:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:34:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:34:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:34:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:34:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:34:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:34:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:34:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:34:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:34:59,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:34:59,254 [236] [DEBUG] [app] Ending request: urn:request:dc831fa7-4717-4a02-9070-709113ab622a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc831fa7-4717-4a02-9070-709113ab622a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:59,254 [237] [DEBUG] [app] Ending request: urn:request:3a33d688-8309-4313-926b-b5660c541fa5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a33d688-8309-4313-926b-b5660c541fa5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:34:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:34:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:34:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 05:34:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:34:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:34:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:34:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:34:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:04.936688+00:00 (in 4.998322 seconds)
exportactionlogsworker stdout | 2025-02-07 05:34:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:59 UTC)" (scheduled at 2025-02-07 05:34:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:34:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:34:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:35:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:35:02,269 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:35:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:35:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:23.420886+00:00 (in 20.001320 seconds)
repositorygcworker stdout | 2025-02-07 05:35:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:03 UTC)" (scheduled at 2025-02-07 05:35:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:35:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:35:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 3, 419852), True, datetime.datetime(2025, 2, 7, 5, 35, 3, 419852), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:35:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:35:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:35:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:35:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:35:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:33.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 05:35:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:33 UTC)" (scheduled at 2025-02-07 05:35:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:35:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 35, 3, 919917), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:35:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:35:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:35:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:35:03,931 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:35:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:35:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:59.937899+00:00 (in 55.000752 seconds)
exportactionlogsworker stdout | 2025-02-07 05:35:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:04 UTC)" (scheduled at 2025-02-07 05:35:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:35:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:35:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 4, 937460), True, datetime.datetime(2025, 2, 7, 5, 35, 4, 937460), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:35:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:35:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:35:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:35:05,113 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:35:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:35:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:18.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 05:35:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:18 UTC)" (scheduled at 2025-02-07 05:35:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:35:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:35:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 8, 732886), True, datetime.datetime(2025, 2, 7, 5, 35, 8, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:35:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:35:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:35:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:35:09,452 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:35:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:35:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:24.423055+00:00 (in 13.998345 seconds)
gcworker stdout | 2025-02-07 05:35:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:40 UTC)" (scheduled at 2025-02-07 05:35:10.424250+00:00)
gcworker stdout | 2025-02-07 05:35:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:35:10,427 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
gcworker stdout | 2025-02-07 05:35:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906210431, None, 1, 0])
gcworker stdout | 2025-02-07 05:35:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:35:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:40 UTC)" executed successfully
autopruneworker stdout | 2025-02-07 05:35:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:35:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:41.831521+00:00 (in 29.999532 seconds)
autopruneworker stdout | 2025-02-07 05:35:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:41 UTC)" (scheduled at 2025-02-07 05:35:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:35:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902911836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:35:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:35:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:35:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:35:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:35:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:29.225378+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2025-02-07 05:35:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:29 UTC)" (scheduled at 2025-02-07 05:35:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:35:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:35:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:35:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:35:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:35:14,235 [238] [DEBUG] [app] Starting request: urn:request:e43d2576-3564-4883-a5f8-089298ff4639 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:14,235 [235] [DEBUG] [app] Starting request: urn:request:d150ce63-6215-4b4e-a4c7-23eb81ce3bd2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:35:14,238 [239] [DEBUG] [app] Starting request: urn:request:a91e8aee-9b0f-4367-a847-e1cbe9c85b20 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:14,238 [246] [DEBUG] [app] Starting request: urn:request:5308cb66-ea45-4f98-b267-7c57971eeb08 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:14,238 [246] [DEBUG] [app] Ending request: urn:request:5308cb66-ea45-4f98-b267-7c57971eeb08 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5308cb66-ea45-4f98-b267-7c57971eeb08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:35:14,238 [239] [DEBUG] [app] Ending request: urn:request:a91e8aee-9b0f-4367-a847-e1cbe9c85b20 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a91e8aee-9b0f-4367-a847-e1cbe9c85b20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:35:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:35:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:14,243 [236] [DEBUG] [app] Starting request: urn:request:512b4504-1b7f-4690-ab8a-a8aa12aacff1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:35:14,243 [237] [DEBUG] [app] Starting request: urn:request:ef0fe001-ce7c-437a-b660-d2067f2f338d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:35:14,243 [236] [DEBUG] [app] Ending request: urn:request:512b4504-1b7f-4690-ab8a-a8aa12aacff1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:512b4504-1b7f-4690-ab8a-a8aa12aacff1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:14,243 [237] [DEBUG] [app] Ending request: urn:request:ef0fe001-ce7c-437a-b660-d2067f2f338d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef0fe001-ce7c-437a-b660-d2067f2f338d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:35:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:14,255 [238] [DEBUG] [app] Ending request: urn:request:e43d2576-3564-4883-a5f8-089298ff4639 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e43d2576-3564-4883-a5f8-089298ff4639', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:14,255 [235] [DEBUG] [app] Ending request: urn:request:d150ce63-6215-4b4e-a4c7-23eb81ce3bd2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d150ce63-6215-4b4e-a4c7-23eb81ce3bd2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:35:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:35:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:35:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:35:14,427 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:35:14,976 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:35:15,579 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:35:17,022 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:35:17,716 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:35:17,813 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:35:17,851 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:35:17,854 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:35:17,857 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:35:17,861 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:35:17,864 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:35:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:35:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:28.732147+00:00 (in 9.999560 seconds)
notificationworker stdout | 2025-02-07 05:35:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:28 UTC)" (scheduled at 2025-02-07 05:35:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:35:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:35:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 18, 732911), True, datetime.datetime(2025, 2, 7, 5, 35, 18, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:35:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:35:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:35:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:35:18,869 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:35:18,872 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:35:18,874 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:35:19,089 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:19,343 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:35:19,833 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:35:19,842 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:35:20,160 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:35:20,309 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:35:20,315 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:35:20,319 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:35:20,322 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:35:20,325 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:35:20,328 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:35:20,330 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:35:20,332 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:35:20,338 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:35:20,505 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:35:20,842 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:35:21,162 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:35:21,234 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:35:22,871 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:35:22,886 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:35:23,303 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:35:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:35:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:03.419110+00:00 (in 39.997761 seconds)
repositorygcworker stdout | 2025-02-07 05:35:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:23 UTC)" (scheduled at 2025-02-07 05:35:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:35:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:35:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:35:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:35:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:40.424250+00:00 (in 16.000697 seconds)
gcworker stdout | 2025-02-07 05:35:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:54 UTC)" (scheduled at 2025-02-07 05:35:24.423055+00:00)
gcworker stdout | 2025-02-07 05:35:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:35:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:35:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:35:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:35:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:38.732147+00:00 (in 9.999585 seconds)
notificationworker stdout | 2025-02-07 05:35:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:38 UTC)" (scheduled at 2025-02-07 05:35:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:35:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:35:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 28, 732831), True, datetime.datetime(2025, 2, 7, 5, 35, 28, 732831), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:35:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:35:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:35:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:35:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:35:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:44.225378+00:00 (in 14.999528 seconds)
quotatotalworker stdout | 2025-02-07 05:35:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:44 UTC)" (scheduled at 2025-02-07 05:35:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:35:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:35:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:35:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:35:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:35:29,235 [238] [DEBUG] [app] Starting request: urn:request:b73605b8-5556-4e18-9e0e-ad00118e4403 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:29,235 [235] [DEBUG] [app] Starting request: urn:request:d05f6ba8-5fc5-43c0-bf41-d1c92b402568 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:35:29,238 [239] [DEBUG] [app] Starting request: urn:request:e2457841-85da-468a-b5de-a0abca879ba0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:29,238 [246] [DEBUG] [app] Starting request: urn:request:989c2881-2eb2-494b-befd-154ec374a735 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:29,238 [239] [DEBUG] [app] Ending request: urn:request:e2457841-85da-468a-b5de-a0abca879ba0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e2457841-85da-468a-b5de-a0abca879ba0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:35:29,238 [246] [DEBUG] [app] Ending request: urn:request:989c2881-2eb2-494b-befd-154ec374a735 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:989c2881-2eb2-494b-befd-154ec374a735', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:35:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:35:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:29,241 [238] [DEBUG] [app] Starting request: urn:request:2b211f6d-6a88-4c42-8524-241cbcc3c2cd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:35:29,241 [237] [DEBUG] [app] Starting request: urn:request:50a42356-629a-4c58-8b92-2f0d1a09a6d0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:35:29,241 [238] [DEBUG] [app] Ending request: urn:request:2b211f6d-6a88-4c42-8524-241cbcc3c2cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2b211f6d-6a88-4c42-8524-241cbcc3c2cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:29,241 [237] [DEBUG] [app] Ending request: urn:request:50a42356-629a-4c58-8b92-2f0d1a09a6d0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:50a42356-629a-4c58-8b92-2f0d1a09a6d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:29,241 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:35:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:29,245 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:29,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:29,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:29,252 [238] [DEBUG] [app] Ending request: urn:request:b73605b8-5556-4e18-9e0e-ad00118e4403 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b73605b8-5556-4e18-9e0e-ad00118e4403', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:29,252 [235] [DEBUG] [app] Ending request: urn:request:d05f6ba8-5fc5-43c0-bf41-d1c92b402568 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d05f6ba8-5fc5-43c0-bf41-d1c92b402568', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:29,252 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:35:29,252 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:35:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:35:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 05:35:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:35:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:32.625945+00:00 (in 1.004390 seconds)
namespacegcworker stdout | 2025-02-07 05:35:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:31 UTC)" (scheduled at 2025-02-07 05:35:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:35:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:35:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 31, 621835), True, datetime.datetime(2025, 2, 7, 5, 35, 31, 621835), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:35:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:35:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:35:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:35:32,305 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:35:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:35:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:31.621131+00:00 (in 58.994745 seconds)
namespacegcworker stdout | 2025-02-07 05:35:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:32 UTC)" (scheduled at 2025-02-07 05:35:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:35:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:35:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:35:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:35:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:03.919286+00:00 (in 29.999558 seconds)
buildlogsarchiver stdout | 2025-02-07 05:35:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:03 UTC)" (scheduled at 2025-02-07 05:35:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:35:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 35, 33, 919911), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:35:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:35:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:35:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:35:33,949 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:35:35,129 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:56.713547+00:00 (in 18.997165 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:37 UTC)" (scheduled at 2025-02-07 05:35:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:35:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:35:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:48.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 05:35:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:48 UTC)" (scheduled at 2025-02-07 05:35:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:35:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:35:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 38, 732861), True, datetime.datetime(2025, 2, 7, 5, 35, 38, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:35:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:35:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:35:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:35:39,470 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:35:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:35:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:54.423055+00:00 (in 13.998364 seconds)
gcworker stdout | 2025-02-07 05:35:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:10 UTC)" (scheduled at 2025-02-07 05:35:40.424250+00:00)
gcworker stdout | 2025-02-07 05:35:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:35:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906240431, None, 1, 0])
gcworker stdout | 2025-02-07 05:35:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:35:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:35:40,437 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:35:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:35:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:11.831521+00:00 (in 29.999486 seconds)
autopruneworker stdout | 2025-02-07 05:35:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:11 UTC)" (scheduled at 2025-02-07 05:35:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:35:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902941836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:35:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:35:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:35:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:35:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:35:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:59.225378+00:00 (in 14.999469 seconds)
quotatotalworker stdout | 2025-02-07 05:35:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:59 UTC)" (scheduled at 2025-02-07 05:35:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:35:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:35:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:35:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:35:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:35:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:35:44,235 [235] [DEBUG] [app] Starting request: urn:request:df0cf425-fe1d-4eb3-bdd2-126883134003 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:44,235 [237] [DEBUG] [app] Starting request: urn:request:68c0f917-8695-40ee-a416-4c887a5f225b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:35:44,238 [243] [DEBUG] [app] Starting request: urn:request:62e27cac-8a4b-41fa-ab6a-66d0b75484a1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:44,238 [246] [DEBUG] [app] Starting request: urn:request:581c1acf-6585-4dac-a5d3-ae46371beee5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:44,239 [243] [DEBUG] [app] Ending request: urn:request:62e27cac-8a4b-41fa-ab6a-66d0b75484a1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:62e27cac-8a4b-41fa-ab6a-66d0b75484a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:35:44,239 [246] [DEBUG] [app] Ending request: urn:request:581c1acf-6585-4dac-a5d3-ae46371beee5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:581c1acf-6585-4dac-a5d3-ae46371beee5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:35:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:35:44,239 [243] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:35:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:44,243 [238] [DEBUG] [app] Starting request: urn:request:6b69cb25-c14e-4ec2-a86b-17511af7eef1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:35:44,243 [236] [DEBUG] [app] Starting request: urn:request:4b313d25-3735-427e-9f25-ff8a59447f8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:35:44,243 [236] [DEBUG] [app] Ending request: urn:request:4b313d25-3735-427e-9f25-ff8a59447f8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4b313d25-3735-427e-9f25-ff8a59447f8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:44,243 [238] [DEBUG] [app] Ending request: urn:request:6b69cb25-c14e-4ec2-a86b-17511af7eef1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6b69cb25-c14e-4ec2-a86b-17511af7eef1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:44,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:35:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:44,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:44,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:44,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:44,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:44,256 [235] [DEBUG] [app] Ending request: urn:request:df0cf425-fe1d-4eb3-bdd2-126883134003 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df0cf425-fe1d-4eb3-bdd2-126883134003', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:44,256 [237] [DEBUG] [app] Ending request: urn:request:68c0f917-8695-40ee-a416-4c887a5f225b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:68c0f917-8695-40ee-a416-4c887a5f225b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:44,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:35:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:35:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:35:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:35:44,464 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:35:45,013 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:35:45,613 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:35:47,050 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:35:47,746 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:35:47,849 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-web stdout | 2025-02-07 05:35:47,860 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:35:47,862 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:35:47,867 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:35:47,870 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:35:47,872 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:35:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:35:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:50.814677+00:00 (in 2.082110 seconds)
notificationworker stdout | 2025-02-07 05:35:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:58 UTC)" (scheduled at 2025-02-07 05:35:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:35:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:35:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 48, 732836), True, datetime.datetime(2025, 2, 7, 5, 35, 48, 732836), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:35:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:35:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:35:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:35:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:35:48,877 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:35:48,880 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:35:48,884 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:35:49,125 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:49,353 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:35:49,859 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:35:49,861 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:35:50,197 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:35:50,320 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:35:50,326 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:35:50,329 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:35:50,332 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:35:50,335 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:35:50,338 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:35:50,342 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:35:50,345 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:35:50,348 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:35:50,524 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:35:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:35:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:35:58.732147+00:00 (in 7.917007 seconds)
notificationworker stdout | 2025-02-07 05:35:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:50 UTC)" (scheduled at 2025-02-07 05:35:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:35:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:35:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:35:50,878 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:35:51,199 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:35:51,271 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:35:52,907 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:35:52,911 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:35:53,333 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:35:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:35:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:10.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 05:35:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:24 UTC)" (scheduled at 2025-02-07 05:35:54.423055+00:00)
gcworker stdout | 2025-02-07 05:35:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:35:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:35:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:35:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:56.118855+00:00 (in 59.999502 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:35:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:56 UTC)" (scheduled at 2025-02-07 05:35:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:35:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:35:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:35:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:37.715878+00:00 (in 41.001894 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:56 UTC)" (scheduled at 2025-02-07 05:35:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:35:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:35:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:35:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:08.732147+00:00 (in 9.999565 seconds)
notificationworker stdout | 2025-02-07 05:35:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:08 UTC)" (scheduled at 2025-02-07 05:35:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:35:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:35:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 35, 58, 732861), True, datetime.datetime(2025, 2, 7, 5, 35, 58, 732861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:35:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:35:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:35:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:35:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:35:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:14.225378+00:00 (in 14.999521 seconds)
quotatotalworker stdout | 2025-02-07 05:35:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:14 UTC)" (scheduled at 2025-02-07 05:35:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:35:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:35:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:35:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:35:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:35:59,235 [237] [DEBUG] [app] Starting request: urn:request:e5df7b1e-9733-44fb-bbbf-eb47b47f6e73 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:59,235 [235] [DEBUG] [app] Starting request: urn:request:f3de9748-a7e2-46ab-841d-9cb2dc2d9fb8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:35:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:59,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:35:59,239 [239] [DEBUG] [app] Starting request: urn:request:ee3c9f5d-a647-4950-8e13-1b7af0c9083b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:59,239 [246] [DEBUG] [app] Starting request: urn:request:895af4da-2499-4607-9d76-e2fb5faf8fac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:35:59,239 [239] [DEBUG] [app] Ending request: urn:request:ee3c9f5d-a647-4950-8e13-1b7af0c9083b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee3c9f5d-a647-4950-8e13-1b7af0c9083b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:35:59,240 [246] [DEBUG] [app] Ending request: urn:request:895af4da-2499-4607-9d76-e2fb5faf8fac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:895af4da-2499-4607-9d76-e2fb5faf8fac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:35:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:35:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:35:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:35:59,242 [237] [DEBUG] [app] Starting request: urn:request:30401b94-7a1a-48d1-8c01-c1503d6a938e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:35:59,242 [237] [DEBUG] [app] Ending request: urn:request:30401b94-7a1a-48d1-8c01-c1503d6a938e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:30401b94-7a1a-48d1-8c01-c1503d6a938e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:59,242 [238] [DEBUG] [app] Starting request: urn:request:b4c3c6e9-1218-4751-a994-5afaf34c49d0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:35:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:59,243 [238] [DEBUG] [app] Ending request: urn:request:b4c3c6e9-1218-4751-a994-5afaf34c49d0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4c3c6e9-1218-4751-a994-5afaf34c49d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:35:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:35:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:35:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:35:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:35:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:35:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:35:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:35:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:35:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:35:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:35:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:35:59,254 [235] [DEBUG] [app] Ending request: urn:request:f3de9748-a7e2-46ab-841d-9cb2dc2d9fb8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f3de9748-a7e2-46ab-841d-9cb2dc2d9fb8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:59,254 [237] [DEBUG] [app] Ending request: urn:request:e5df7b1e-9733-44fb-bbbf-eb47b47f6e73 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5df7b1e-9733-44fb-bbbf-eb47b47f6e73', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:35:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:35:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:35:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:35:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:35:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:35:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:35:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:35:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:04.936688+00:00 (in 4.998314 seconds)
exportactionlogsworker stdout | 2025-02-07 05:35:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:59 UTC)" (scheduled at 2025-02-07 05:35:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:35:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:35:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:36:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:36:02,341 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:36:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:36:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:23.420886+00:00 (in 20.001320 seconds)
repositorygcworker stdout | 2025-02-07 05:36:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:03 UTC)" (scheduled at 2025-02-07 05:36:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:36:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:36:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 3, 419843), True, datetime.datetime(2025, 2, 7, 5, 36, 3, 419843), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:36:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:36:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:36:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:36:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:36:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:33.919286+00:00 (in 29.999555 seconds)
buildlogsarchiver stdout | 2025-02-07 05:36:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:33 UTC)" (scheduled at 2025-02-07 05:36:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:36:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 36, 3, 919915), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:36:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:36:03,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:36:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:36:03,975 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:36:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:36:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:59.937899+00:00 (in 55.000793 seconds)
exportactionlogsworker stdout | 2025-02-07 05:36:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:04 UTC)" (scheduled at 2025-02-07 05:36:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:36:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:36:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 4, 937402), True, datetime.datetime(2025, 2, 7, 5, 36, 4, 937402), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:36:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:36:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:36:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:36:05,164 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:36:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:36:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:18.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 05:36:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:18 UTC)" (scheduled at 2025-02-07 05:36:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:36:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:36:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 8, 732857), True, datetime.datetime(2025, 2, 7, 5, 36, 8, 732857), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:36:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:36:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:36:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:36:09,483 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:36:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:36:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:24.423055+00:00 (in 13.998383 seconds)
gcworker stdout | 2025-02-07 05:36:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:40 UTC)" (scheduled at 2025-02-07 05:36:10.424250+00:00)
gcworker stdout | 2025-02-07 05:36:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:36:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906270430, None, 1, 0])
gcworker stdout | 2025-02-07 05:36:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:36:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:36:10,471 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:36:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:36:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:41.831521+00:00 (in 29.999499 seconds)
autopruneworker stdout | 2025-02-07 05:36:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:41 UTC)" (scheduled at 2025-02-07 05:36:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:36:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738902971836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:36:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:36:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:36:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:36:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:36:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:29.225378+00:00 (in 14.999576 seconds)
quotatotalworker stdout | 2025-02-07 05:36:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:29 UTC)" (scheduled at 2025-02-07 05:36:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:36:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:36:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:36:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:36:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:36:14,235 [235] [DEBUG] [app] Starting request: urn:request:8ec019f5-e77b-4ecc-a92f-28c06fcd88e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:14,235 [238] [DEBUG] [app] Starting request: urn:request:bf7667e8-646a-44eb-90f8-9dc30c6fd82d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:36:14,238 [239] [DEBUG] [app] Starting request: urn:request:c8dead1e-52cd-4f43-89fc-d36be2aa3bf4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:36:14,238 [244] [DEBUG] [app] Starting request: urn:request:47705154-b7b3-484d-ad20-86b860e27395 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:36:14,239 [239] [DEBUG] [app] Ending request: urn:request:c8dead1e-52cd-4f43-89fc-d36be2aa3bf4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c8dead1e-52cd-4f43-89fc-d36be2aa3bf4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:36:14,239 [244] [DEBUG] [app] Ending request: urn:request:47705154-b7b3-484d-ad20-86b860e27395 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:47705154-b7b3-484d-ad20-86b860e27395', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:36:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:36:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:36:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:14,242 [237] [DEBUG] [app] Starting request: urn:request:4f716cdb-cedd-4eef-9f4b-9342f06340da (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:14,242 [235] [DEBUG] [app] Starting request: urn:request:89ca0e22-6b7a-46fd-90ef-423d005d4a69 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:14,242 [235] [DEBUG] [app] Ending request: urn:request:89ca0e22-6b7a-46fd-90ef-423d005d4a69 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89ca0e22-6b7a-46fd-90ef-423d005d4a69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:14,242 [237] [DEBUG] [app] Ending request: urn:request:4f716cdb-cedd-4eef-9f4b-9342f06340da (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f716cdb-cedd-4eef-9f4b-9342f06340da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:36:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:36:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:36:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:36:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:14,253 [235] [DEBUG] [app] Ending request: urn:request:8ec019f5-e77b-4ecc-a92f-28c06fcd88e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ec019f5-e77b-4ecc-a92f-28c06fcd88e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:36:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:36:14,254 [238] [DEBUG] [app] Ending request: urn:request:bf7667e8-646a-44eb-90f8-9dc30c6fd82d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bf7667e8-646a-44eb-90f8-9dc30c6fd82d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:36:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:36:14,493 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:36:15,049 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:36:15,633 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:36:17,079 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:36:17,782 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gunicorn-web stdout | 2025-02-07 05:36:17,867 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:36:17,871 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:36:17,874 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:36:17,878 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:36:17,881 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
chunkcleanupworker stdout | 2025-02-07 05:36:17,885 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:36:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:36:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:28.732147+00:00 (in 9.999587 seconds)
notificationworker stdout | 2025-02-07 05:36:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:28 UTC)" (scheduled at 2025-02-07 05:36:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:36:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:36:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 18, 732874), True, datetime.datetime(2025, 2, 7, 5, 36, 18, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:36:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:36:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:36:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:36:18,886 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:36:18,890 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:36:18,893 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:36:19,157 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:19,389 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:36:19,887 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:36:19,890 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:36:20,211 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:36:20,330 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:36:20,334 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:36:20,336 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:36:20,343 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:36:20,346 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:36:20,350 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:36:20,356 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:36:20,358 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:36:20,361 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:36:20,561 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:36:20,902 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:36:21,211 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:36:21,306 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:36:22,936 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:36:22,942 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:36:23,342 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:36:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:36:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:03.419110+00:00 (in 39.997760 seconds)
repositorygcworker stdout | 2025-02-07 05:36:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:23 UTC)" (scheduled at 2025-02-07 05:36:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:36:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:36:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:36:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:36:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:40.424250+00:00 (in 16.000741 seconds)
gcworker stdout | 2025-02-07 05:36:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:54 UTC)" (scheduled at 2025-02-07 05:36:24.423055+00:00)
gcworker stdout | 2025-02-07 05:36:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:36:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:36:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:36:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:36:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:38.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 05:36:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:38 UTC)" (scheduled at 2025-02-07 05:36:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:36:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:36:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 28, 732914), True, datetime.datetime(2025, 2, 7, 5, 36, 28, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:36:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:36:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:36:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:36:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:36:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:44.225378+00:00 (in 14.999566 seconds)
quotatotalworker stdout | 2025-02-07 05:36:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:44 UTC)" (scheduled at 2025-02-07 05:36:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:36:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:36:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:36:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:36:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:36:29,236 [237] [DEBUG] [app] Starting request: urn:request:29714d36-5fde-4b9e-b3fd-70060d8e887b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:29,236 [235] [DEBUG] [app] Starting request: urn:request:6478c109-5ae5-4edc-9c01-07da4612b8ab (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:36:29,239 [239] [DEBUG] [app] Starting request: urn:request:ab77f705-ac12-4c00-9b6a-fc4d82aa5944 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:36:29,239 [246] [DEBUG] [app] Starting request: urn:request:2049e671-1292-4a53-a95b-43595e35ee14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:36:29,239 [239] [DEBUG] [app] Ending request: urn:request:ab77f705-ac12-4c00-9b6a-fc4d82aa5944 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab77f705-ac12-4c00-9b6a-fc4d82aa5944', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:36:29,239 [246] [DEBUG] [app] Ending request: urn:request:2049e671-1292-4a53-a95b-43595e35ee14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2049e671-1292-4a53-a95b-43595e35ee14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:36:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:36:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:36:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:29,242 [237] [DEBUG] [app] Starting request: urn:request:051e6d3e-cc89-47d7-987e-f53af8eaa4dd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:29,242 [237] [DEBUG] [app] Ending request: urn:request:051e6d3e-cc89-47d7-987e-f53af8eaa4dd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:051e6d3e-cc89-47d7-987e-f53af8eaa4dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:36:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:29,243 [238] [DEBUG] [app] Starting request: urn:request:59f56d96-e1cf-445c-a1d0-059e1889f524 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:36:29,243 [238] [DEBUG] [app] Ending request: urn:request:59f56d96-e1cf-445c-a1d0-059e1889f524 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59f56d96-e1cf-445c-a1d0-059e1889f524', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:36:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:36:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:36:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:29,253 [237] [DEBUG] [app] Ending request: urn:request:29714d36-5fde-4b9e-b3fd-70060d8e887b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29714d36-5fde-4b9e-b3fd-70060d8e887b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:36:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:36:29,254 [235] [DEBUG] [app] Ending request: urn:request:6478c109-5ae5-4edc-9c01-07da4612b8ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6478c109-5ae5-4edc-9c01-07da4612b8ab', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:36:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 05:36:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:36:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:32.625945+00:00 (in 1.004361 seconds)
namespacegcworker stdout | 2025-02-07 05:36:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:31 UTC)" (scheduled at 2025-02-07 05:36:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:36:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:36:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 31, 621865), True, datetime.datetime(2025, 2, 7, 5, 36, 31, 621865), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:36:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:36:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:36:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:36:32,377 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:36:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:36:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:13.624301+00:00 (in 40.997905 seconds)
namespacegcworker stdout | 2025-02-07 05:36:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:32 UTC)" (scheduled at 2025-02-07 05:36:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:36:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:36:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:36:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:36:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:03.919286+00:00 (in 29.999509 seconds)
buildlogsarchiver stdout | 2025-02-07 05:36:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:03 UTC)" (scheduled at 2025-02-07 05:36:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:36:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 36, 33, 919963), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:36:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:36:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:36:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:36:34,011 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:36:35,190 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:56.713547+00:00 (in 18.997201 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:37 UTC)" (scheduled at 2025-02-07 05:36:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:36:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:36:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:48.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 05:36:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:48 UTC)" (scheduled at 2025-02-07 05:36:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:36:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:36:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 38, 732923), True, datetime.datetime(2025, 2, 7, 5, 36, 38, 732923), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:36:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:36:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:36:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:36:39,495 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:36:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:36:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:54.423055+00:00 (in 13.998385 seconds)
gcworker stdout | 2025-02-07 05:36:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:10 UTC)" (scheduled at 2025-02-07 05:36:40.424250+00:00)
gcworker stdout | 2025-02-07 05:36:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:36:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906300431, None, 1, 0])
gcworker stdout | 2025-02-07 05:36:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:36:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:36:40,505 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:36:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:36:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:11.831521+00:00 (in 29.999536 seconds)
autopruneworker stdout | 2025-02-07 05:36:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:11 UTC)" (scheduled at 2025-02-07 05:36:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:36:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903001836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:36:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:36:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:36:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:36:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:36:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:59.225378+00:00 (in 14.999568 seconds)
quotatotalworker stdout | 2025-02-07 05:36:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:59 UTC)" (scheduled at 2025-02-07 05:36:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:36:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:36:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:36:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:36:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:36:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:36:44,235 [235] [DEBUG] [app] Starting request: urn:request:2effcfd2-4e60-4327-bbbd-ec3b08183745 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:44,235 [236] [DEBUG] [app] Starting request: urn:request:19441a95-571e-479f-be93-5ce5b52f7b53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:36:44,238 [246] [DEBUG] [app] Starting request: urn:request:4d34a193-e74e-4f37-a014-10961bcd4868 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:36:44,238 [239] [DEBUG] [app] Starting request: urn:request:f7b5c869-2e18-4855-8d6f-3214871fbb24 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:36:44,239 [246] [DEBUG] [app] Ending request: urn:request:4d34a193-e74e-4f37-a014-10961bcd4868 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d34a193-e74e-4f37-a014-10961bcd4868', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:36:44,239 [239] [DEBUG] [app] Ending request: urn:request:f7b5c869-2e18-4855-8d6f-3214871fbb24 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f7b5c869-2e18-4855-8d6f-3214871fbb24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:36:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:36:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:36:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:44,242 [236] [DEBUG] [app] Starting request: urn:request:f759152c-68d1-403c-90ed-24136e15a67d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:44,242 [236] [DEBUG] [app] Ending request: urn:request:f759152c-68d1-403c-90ed-24136e15a67d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f759152c-68d1-403c-90ed-24136e15a67d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:36:44,242 [238] [DEBUG] [app] Starting request: urn:request:2e75c417-2ffd-4a77-b154-eb27a639d4f1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:44,243 [238] [DEBUG] [app] Ending request: urn:request:2e75c417-2ffd-4a77-b154-eb27a639d4f1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e75c417-2ffd-4a77-b154-eb27a639d4f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:36:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:36:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:36:44,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:44,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:44,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:44,253 [236] [DEBUG] [app] Ending request: urn:request:19441a95-571e-479f-be93-5ce5b52f7b53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:19441a95-571e-479f-be93-5ce5b52f7b53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:36:44,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 05:36:44,254 [235] [DEBUG] [app] Ending request: urn:request:2effcfd2-4e60-4327-bbbd-ec3b08183745 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2effcfd2-4e60-4327-bbbd-ec3b08183745', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:36:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:36:44,530 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:36:45,085 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:36:45,666 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:36:47,115 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:36:47,819 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gunicorn-web stdout | 2025-02-07 05:36:47,875 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:36:47,878 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:36:47,881 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:36:47,884 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:36:47,888 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
chunkcleanupworker stdout | 2025-02-07 05:36:47,920 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:36:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:36:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:50.814677+00:00 (in 2.082102 seconds)
notificationworker stdout | 2025-02-07 05:36:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:58 UTC)" (scheduled at 2025-02-07 05:36:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:36:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:36:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 48, 732845), True, datetime.datetime(2025, 2, 7, 5, 36, 48, 732845), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:36:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:36:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:36:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:36:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:36:48,894 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:36:48,898 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:36:48,901 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:36:49,193 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:49,425 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:36:49,901 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:36:49,922 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:36:50,244 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:36:50,340 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:36:50,343 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:36:50,346 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:36:50,352 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:36:50,358 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:36:50,361 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:36:50,366 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:36:50,368 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:36:50,372 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:36:50,589 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:36:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:36:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:36:58.732147+00:00 (in 7.917009 seconds)
notificationworker stdout | 2025-02-07 05:36:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:50 UTC)" (scheduled at 2025-02-07 05:36:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:36:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:36:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:36:50,936 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:36:51,247 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:36:51,332 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:36:52,951 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:36:52,972 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:36:53,378 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:36:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:36:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:10.424250+00:00 (in 16.000761 seconds)
gcworker stdout | 2025-02-07 05:36:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:24 UTC)" (scheduled at 2025-02-07 05:36:54.423055+00:00)
gcworker stdout | 2025-02-07 05:36:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:36:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:36:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:36:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:56.118855+00:00 (in 59.999493 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:36:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:56 UTC)" (scheduled at 2025-02-07 05:36:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:36:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:36:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:36:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:37.715878+00:00 (in 41.001886 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:56 UTC)" (scheduled at 2025-02-07 05:36:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:36:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:36:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:36:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:08.732147+00:00 (in 9.999586 seconds)
notificationworker stdout | 2025-02-07 05:36:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:08 UTC)" (scheduled at 2025-02-07 05:36:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:36:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:36:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 36, 58, 732827), True, datetime.datetime(2025, 2, 7, 5, 36, 58, 732827), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:36:58,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:36:58,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:36:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:36:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:36:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:14.225378+00:00 (in 14.999552 seconds)
quotatotalworker stdout | 2025-02-07 05:36:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:14 UTC)" (scheduled at 2025-02-07 05:36:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:36:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:36:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:36:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:36:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:36:59,236 [236] [DEBUG] [app] Starting request: urn:request:21b76446-0b9c-49ed-840e-e6670a47e0d1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:59,236 [235] [DEBUG] [app] Starting request: urn:request:7e7ad1c1-a8dc-4f11-881f-fc9e3b99815f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:36:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:36:59,239 [239] [DEBUG] [app] Starting request: urn:request:40c7c998-861e-4d4d-a89e-88cf0618f81c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:36:59,239 [239] [DEBUG] [app] Ending request: urn:request:40c7c998-861e-4d4d-a89e-88cf0618f81c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40c7c998-861e-4d4d-a89e-88cf0618f81c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:36:59,239 [246] [DEBUG] [app] Starting request: urn:request:046ee245-d8f8-4d0d-ba91-abbb08f722cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:36:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:36:59,239 [246] [DEBUG] [app] Ending request: urn:request:046ee245-d8f8-4d0d-ba91-abbb08f722cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:046ee245-d8f8-4d0d-ba91-abbb08f722cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:59,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:36:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:36:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:36:59,242 [238] [DEBUG] [app] Starting request: urn:request:d3e9e768-093e-4ad3-912b-0e6c0cbef827 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:59,242 [238] [DEBUG] [app] Ending request: urn:request:d3e9e768-093e-4ad3-912b-0e6c0cbef827 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d3e9e768-093e-4ad3-912b-0e6c0cbef827', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:36:59,243 [236] [DEBUG] [app] Starting request: urn:request:89c93b7c-1633-4d66-b2eb-9115b8dbe27e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:36:59,243 [236] [DEBUG] [app] Ending request: urn:request:89c93b7c-1633-4d66-b2eb-9115b8dbe27e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:89c93b7c-1633-4d66-b2eb-9115b8dbe27e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:36:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:36:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:36:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:36:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:36:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:36:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:36:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:36:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:36:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:36:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:36:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:36:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:36:59,255 [236] [DEBUG] [app] Ending request: urn:request:21b76446-0b9c-49ed-840e-e6670a47e0d1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:21b76446-0b9c-49ed-840e-e6670a47e0d1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:59,255 [235] [DEBUG] [app] Ending request: urn:request:7e7ad1c1-a8dc-4f11-881f-fc9e3b99815f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e7ad1c1-a8dc-4f11-881f-fc9e3b99815f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:36:59,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:36:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:36:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:36:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:36:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:36:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:36:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:36:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:04.936688+00:00 (in 4.998342 seconds)
exportactionlogsworker stdout | 2025-02-07 05:36:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:59 UTC)" (scheduled at 2025-02-07 05:36:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:36:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:36:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:37:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:37:02,410 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:37:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:37:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:23.420886+00:00 (in 20.001285 seconds)
repositorygcworker stdout | 2025-02-07 05:37:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:03 UTC)" (scheduled at 2025-02-07 05:37:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:37:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:37:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 3, 419888), True, datetime.datetime(2025, 2, 7, 5, 37, 3, 419888), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:37:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:37:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:37:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:37:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:37:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:33.919286+00:00 (in 29.999517 seconds)
buildlogsarchiver stdout | 2025-02-07 05:37:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:33 UTC)" (scheduled at 2025-02-07 05:37:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:37:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 37, 3, 919964), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:37:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:37:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:37:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:37:04,048 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:37:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:37:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:59.937899+00:00 (in 55.000766 seconds)
exportactionlogsworker stdout | 2025-02-07 05:37:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:04 UTC)" (scheduled at 2025-02-07 05:37:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:37:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:37:04,997 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 4, 937435), True, datetime.datetime(2025, 2, 7, 5, 37, 4, 937435), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:37:05,004 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:37:05,004 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:37:05,004 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:37:05,227 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:37:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:18.732147+00:00 (in 9.999572 seconds)
notificationworker stdout | 2025-02-07 05:37:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:18 UTC)" (scheduled at 2025-02-07 05:37:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:37:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:37:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 8, 732851), True, datetime.datetime(2025, 2, 7, 5, 37, 8, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:37:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:37:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:37:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:37:09,511 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:37:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:37:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:24.423055+00:00 (in 13.998378 seconds)
gcworker stdout | 2025-02-07 05:37:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:40 UTC)" (scheduled at 2025-02-07 05:37:10.424250+00:00)
gcworker stdout | 2025-02-07 05:37:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:37:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906330431, None, 1, 0])
gcworker stdout | 2025-02-07 05:37:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:37:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:37:10,542 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:37:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:37:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:41.831521+00:00 (in 29.999491 seconds)
autopruneworker stdout | 2025-02-07 05:37:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:41 UTC)" (scheduled at 2025-02-07 05:37:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:37:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903031836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:37:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:37:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:37:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:41 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:37:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:37:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:31.621131+00:00 (in 17.996399 seconds)
namespacegcworker stdout | 2025-02-07 05:37:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:42:13 UTC)" (scheduled at 2025-02-07 05:37:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:37:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 37, 13, 624975), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:37:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 37, 13, 624975), True, datetime.datetime(2025, 2, 7, 5, 37, 13, 624975), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:37:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 37, 13, 624975), True, datetime.datetime(2025, 2, 7, 5, 37, 13, 624975), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 37, 13, 624975), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:37:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:37:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:42:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:37:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:37:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:29.225378+00:00 (in 14.999555 seconds)
quotatotalworker stdout | 2025-02-07 05:37:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:29 UTC)" (scheduled at 2025-02-07 05:37:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:37:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:37:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:37:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:37:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:37:14,235 [235] [DEBUG] [app] Starting request: urn:request:0b1ac903-26e0-49ae-829f-b63dfbce8c70 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:14,235 [237] [DEBUG] [app] Starting request: urn:request:4ea52593-98d4-47d9-97f1-dcaa9655efc4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:37:14,239 [239] [DEBUG] [app] Starting request: urn:request:bbb44133-e2e6-4ba8-9579-daf439934d9f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:14,239 [244] [DEBUG] [app] Starting request: urn:request:fda26e9a-cc6b-4738-bcf1-3db746e3277b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:14,239 [239] [DEBUG] [app] Ending request: urn:request:bbb44133-e2e6-4ba8-9579-daf439934d9f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bbb44133-e2e6-4ba8-9579-daf439934d9f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:37:14,239 [244] [DEBUG] [app] Ending request: urn:request:fda26e9a-cc6b-4738-bcf1-3db746e3277b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fda26e9a-cc6b-4738-bcf1-3db746e3277b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:37:14,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:37:14,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:37:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:14,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:14,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:14,243 [236] [DEBUG] [app] Starting request: urn:request:cf1527bd-c22c-469e-9c6c-489c61566c0a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:37:14,243 [238] [DEBUG] [app] Starting request: urn:request:f76c4c2f-6421-4b20-a771-2ae7f4344740 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:37:14,244 [236] [DEBUG] [app] Ending request: urn:request:cf1527bd-c22c-469e-9c6c-489c61566c0a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cf1527bd-c22c-469e-9c6c-489c61566c0a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:37:14,244 [238] [DEBUG] [app] Ending request: urn:request:f76c4c2f-6421-4b20-a771-2ae7f4344740 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f76c4c2f-6421-4b20-a771-2ae7f4344740', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:37:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:14,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:14,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:14,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:14,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:37:14,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:37:14,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:14,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:14,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:14,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:14,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:14,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:14,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:14,256 [237] [DEBUG] [app] Ending request: urn:request:4ea52593-98d4-47d9-97f1-dcaa9655efc4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ea52593-98d4-47d9-97f1-dcaa9655efc4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:14,256 [235] [DEBUG] [app] Ending request: urn:request:0b1ac903-26e0-49ae-829f-b63dfbce8c70 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b1ac903-26e0-49ae-829f-b63dfbce8c70', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:37:14,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:37:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:37:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
quotatotalworker stdout | 2025-02-07 05:37:14,545 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:37:15,122 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:37:15,695 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:37:17,149 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:37:17,856 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gunicorn-web stdout | 2025-02-07 05:37:17,883 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:37:17,887 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:37:17,890 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:37:17,894 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:37:17,897 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
chunkcleanupworker stdout | 2025-02-07 05:37:17,931 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:37:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:28.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 05:37:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:28 UTC)" (scheduled at 2025-02-07 05:37:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:37:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:37:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 18, 732882), True, datetime.datetime(2025, 2, 7, 5, 37, 18, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:37:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:37:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:37:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:37:18,902 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:37:18,905 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:37:18,908 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:37:19,229 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:19,435 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:37:19,932 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:37:19,935 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:37:20,270 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:37:20,348 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:37:20,353 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:37:20,356 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:37:20,361 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:37:20,369 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:37:20,372 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:37:20,375 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:37:20,378 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:37:20,380 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:37:20,624 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:37:20,970 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:37:21,257 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:37:21,344 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:37:22,988 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:37:23,008 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:37:23,403 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
repositorygcworker stdout | 2025-02-07 05:37:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:37:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:03.419110+00:00 (in 39.997758 seconds)
repositorygcworker stdout | 2025-02-07 05:37:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:23 UTC)" (scheduled at 2025-02-07 05:37:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:37:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:37:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:23 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:37:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:37:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:40.424250+00:00 (in 16.000728 seconds)
gcworker stdout | 2025-02-07 05:37:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:54 UTC)" (scheduled at 2025-02-07 05:37:24.423055+00:00)
gcworker stdout | 2025-02-07 05:37:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:37:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:37:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:37:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:33.814166+00:00 (in 5.081590 seconds)
notificationworker stdout | 2025-02-07 05:37:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:38 UTC)" (scheduled at 2025-02-07 05:37:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:37:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:37:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 28, 732843), True, datetime.datetime(2025, 2, 7, 5, 37, 28, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:37:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:37:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:37:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:37:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:37:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:44.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 05:37:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:44 UTC)" (scheduled at 2025-02-07 05:37:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:37:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:37:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:37:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:37:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:37:29,235 [238] [DEBUG] [app] Starting request: urn:request:ccad01c3-2d71-4bca-8a38-63251ad1e784 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:29,235 [237] [DEBUG] [app] Starting request: urn:request:ec878503-c8d3-4701-aae8-cb84d5be9688 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:37:29,239 [239] [DEBUG] [app] Starting request: urn:request:89a68269-0a3b-4814-81b4-03441e5588c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:29,239 [246] [DEBUG] [app] Starting request: urn:request:06b06ab8-be72-4025-85a1-0574ae593705 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:29,239 [246] [DEBUG] [app] Ending request: urn:request:06b06ab8-be72-4025-85a1-0574ae593705 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:06b06ab8-be72-4025-85a1-0574ae593705', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:37:29,239 [239] [DEBUG] [app] Ending request: urn:request:89a68269-0a3b-4814-81b4-03441e5588c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:89a68269-0a3b-4814-81b4-03441e5588c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:37:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:37:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:37:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:29,242 [238] [DEBUG] [app] Starting request: urn:request:979b8097-950b-4fb2-9724-e4659e817db3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:37:29,242 [238] [DEBUG] [app] Ending request: urn:request:979b8097-950b-4fb2-9724-e4659e817db3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:979b8097-950b-4fb2-9724-e4659e817db3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:37:29,242 [235] [DEBUG] [app] Starting request: urn:request:2561b59a-2b1f-4ded-9c0d-c3cb9b9d8aeb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:37:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:37:29,243 [235] [DEBUG] [app] Ending request: urn:request:2561b59a-2b1f-4ded-9c0d-c3cb9b9d8aeb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2561b59a-2b1f-4ded-9c0d-c3cb9b9d8aeb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:37:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:37:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:37:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:29,254 [238] [DEBUG] [app] Ending request: urn:request:ccad01c3-2d71-4bca-8a38-63251ad1e784 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ccad01c3-2d71-4bca-8a38-63251ad1e784', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:29,254 [237] [DEBUG] [app] Ending request: urn:request:ec878503-c8d3-4701-aae8-cb84d5be9688 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ec878503-c8d3-4701-aae8-cb84d5be9688', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:37:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:37:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:37:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
namespacegcworker stdout | 2025-02-07 05:37:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:37:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:32.625945+00:00 (in 1.004351 seconds)
namespacegcworker stdout | 2025-02-07 05:37:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:31 UTC)" (scheduled at 2025-02-07 05:37:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:37:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:37:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 31, 621861), True, datetime.datetime(2025, 2, 7, 5, 37, 31, 621861), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:37:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:37:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:37:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:37:32,435 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:37:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:37:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:31.621131+00:00 (in 58.994739 seconds)
namespacegcworker stdout | 2025-02-07 05:37:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:32 UTC)" (scheduled at 2025-02-07 05:37:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:37:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:37:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:32 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:37:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:38.732147+00:00 (in 4.917537 seconds)
notificationworker stdout | 2025-02-07 05:37:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:42:33 UTC)" (scheduled at 2025-02-07 05:37:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:37:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 37, 33, 814880), 'notification/%'])
notificationworker stdout | 2025-02-07 05:37:33,822 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 37, 33, 814880), True, datetime.datetime(2025, 2, 7, 5, 37, 33, 814880), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:37:33,824 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 37, 33, 814880), True, datetime.datetime(2025, 2, 7, 5, 37, 33, 814880), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 37, 33, 814880), 'notification/%'])
notificationworker stdout | 2025-02-07 05:37:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:37:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:42:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:37:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:37:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:03.919286+00:00 (in 29.999502 seconds)
buildlogsarchiver stdout | 2025-02-07 05:37:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:03 UTC)" (scheduled at 2025-02-07 05:37:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:37:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 37, 33, 919973), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:37:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:37:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:37:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:37:34,084 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:37:35,263 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:56.713547+00:00 (in 18.997233 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:37 UTC)" (scheduled at 2025-02-07 05:37:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:37:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:48.732147+00:00 (in 9.999566 seconds)
notificationworker stdout | 2025-02-07 05:37:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:48 UTC)" (scheduled at 2025-02-07 05:37:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:37:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:37:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 38, 732866), True, datetime.datetime(2025, 2, 7, 5, 37, 38, 732866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:37:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:37:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:37:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:37:39,526 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:37:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:37:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:54.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 05:37:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:10 UTC)" (scheduled at 2025-02-07 05:37:40.424250+00:00)
gcworker stdout | 2025-02-07 05:37:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:37:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906360431, None, 1, 0])
gcworker stdout | 2025-02-07 05:37:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:37:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:37:40,578 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:37:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:37:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:11.831521+00:00 (in 29.999510 seconds)
autopruneworker stdout | 2025-02-07 05:37:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:11 UTC)" (scheduled at 2025-02-07 05:37:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:37:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903061836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:37:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:37:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:37:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:37:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:37:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:59.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 05:37:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:59 UTC)" (scheduled at 2025-02-07 05:37:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:37:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:37:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:37:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:37:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:37:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:37:44,236 [238] [DEBUG] [app] Starting request: urn:request:167ef9ad-1f5a-4de4-84db-5295a1d84f0d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:44,236 [237] [DEBUG] [app] Starting request: urn:request:5b08d9ae-0aff-453c-95a3-c88dc9d4f7fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:37:44,239 [239] [DEBUG] [app] Starting request: urn:request:da4337c6-9d2a-40a2-91da-b7c1b3059f59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:44,239 [245] [DEBUG] [app] Starting request: urn:request:28d7b6b2-e8b9-4792-9986-286db92b9cc4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:44,239 [239] [DEBUG] [app] Ending request: urn:request:da4337c6-9d2a-40a2-91da-b7c1b3059f59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:da4337c6-9d2a-40a2-91da-b7c1b3059f59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:37:44,239 [245] [DEBUG] [app] Ending request: urn:request:28d7b6b2-e8b9-4792-9986-286db92b9cc4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:28d7b6b2-e8b9-4792-9986-286db92b9cc4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:37:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:37:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:44,242 [238] [DEBUG] [app] Starting request: urn:request:c77c3a55-8d96-4c44-a4af-7b54f52266f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:37:44,242 [238] [DEBUG] [app] Ending request: urn:request:c77c3a55-8d96-4c44-a4af-7b54f52266f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c77c3a55-8d96-4c44-a4af-7b54f52266f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:37:44,242 [236] [DEBUG] [app] Starting request: urn:request:b4f43ece-c56f-4516-b4b0-3ccd9c28f8e7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:37:44,242 [236] [DEBUG] [app] Ending request: urn:request:b4f43ece-c56f-4516-b4b0-3ccd9c28f8e7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4f43ece-c56f-4516-b4b0-3ccd9c28f8e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:37:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:37:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:37:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:37:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:44,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:44,253 [237] [DEBUG] [app] Ending request: urn:request:5b08d9ae-0aff-453c-95a3-c88dc9d4f7fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5b08d9ae-0aff-453c-95a3-c88dc9d4f7fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:44,253 [238] [DEBUG] [app] Ending request: urn:request:167ef9ad-1f5a-4de4-84db-5295a1d84f0d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:167ef9ad-1f5a-4de4-84db-5295a1d84f0d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:37:44,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:37:44,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:37:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
quotatotalworker stdout | 2025-02-07 05:37:44,578 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:37:45,146 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:37:45,731 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:37:47,162 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
notificationworker stdout | 2025-02-07 05:37:47,887 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
gunicorn-web stdout | 2025-02-07 05:37:47,892 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:37:47,895 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:37:47,898 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:37:47,900 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:37:47,903 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
chunkcleanupworker stdout | 2025-02-07 05:37:47,952 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:37:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:50.814677+00:00 (in 2.082089 seconds)
notificationworker stdout | 2025-02-07 05:37:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:58 UTC)" (scheduled at 2025-02-07 05:37:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:37:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:37:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 48, 732863), True, datetime.datetime(2025, 2, 7, 5, 37, 48, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:37:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:37:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:37:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:37:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:37:48,911 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:37:48,913 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:37:48,916 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:37:49,265 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:49,469 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:37:49,946 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:37:49,969 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:37:50,291 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:37:50,356 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:37:50,363 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:37:50,366 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:37:50,369 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:37:50,379 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:37:50,382 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:37:50,385 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:37:50,387 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:37:50,391 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:37:50,661 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:37:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:37:58.732147+00:00 (in 7.917024 seconds)
notificationworker stdout | 2025-02-07 05:37:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:50 UTC)" (scheduled at 2025-02-07 05:37:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:37:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:37:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:37:51,006 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:37:51,275 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:37:51,351 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:37:53,005 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:37:53,044 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:37:53,431 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:37:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:37:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:10.424250+00:00 (in 16.000681 seconds)
gcworker stdout | 2025-02-07 05:37:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:24 UTC)" (scheduled at 2025-02-07 05:37:54.423055+00:00)
gcworker stdout | 2025-02-07 05:37:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:37:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:37:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:37:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:56.118855+00:00 (in 59.999475 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:37:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:56 UTC)" (scheduled at 2025-02-07 05:37:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:37:56,120 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:37:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:37:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:37.715878+00:00 (in 41.001881 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:56 UTC)" (scheduled at 2025-02-07 05:37:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:37:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:37:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:37:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:08.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 05:37:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:08 UTC)" (scheduled at 2025-02-07 05:37:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:37:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:37:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 37, 58, 732924), True, datetime.datetime(2025, 2, 7, 5, 37, 58, 732924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:37:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:37:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:37:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:37:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:37:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:14.225378+00:00 (in 14.999519 seconds)
quotatotalworker stdout | 2025-02-07 05:37:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:14 UTC)" (scheduled at 2025-02-07 05:37:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:37:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:37:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:37:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:37:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:37:59,236 [237] [DEBUG] [app] Starting request: urn:request:3edea5b1-190e-41b8-96c3-13da8c371ecb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:59,236 [235] [DEBUG] [app] Starting request: urn:request:1edef679-e8c8-4cf7-bb9b-33f8c6d55b21 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:37:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:37:59,239 [246] [DEBUG] [app] Starting request: urn:request:1defd994-97fd-4705-b013-10d1c9ab54dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:59,239 [239] [DEBUG] [app] Starting request: urn:request:5dc84197-4d65-4e47-8343-ffb76034207d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:37:59,239 [246] [DEBUG] [app] Ending request: urn:request:1defd994-97fd-4705-b013-10d1c9ab54dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1defd994-97fd-4705-b013-10d1c9ab54dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:37:59,239 [239] [DEBUG] [app] Ending request: urn:request:5dc84197-4d65-4e47-8343-ffb76034207d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5dc84197-4d65-4e47-8343-ffb76034207d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:37:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:37:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:37:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:37:59,242 [237] [DEBUG] [app] Starting request: urn:request:72f8a6f5-d171-4fa4-a880-ab15ccd88884 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:37:59,242 [236] [DEBUG] [app] Starting request: urn:request:0fa5ef1b-091c-4cc2-b3db-ec153a3038b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:37:59,243 [237] [DEBUG] [app] Ending request: urn:request:72f8a6f5-d171-4fa4-a880-ab15ccd88884 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:72f8a6f5-d171-4fa4-a880-ab15ccd88884', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:37:59,243 [236] [DEBUG] [app] Ending request: urn:request:0fa5ef1b-091c-4cc2-b3db-ec153a3038b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0fa5ef1b-091c-4cc2-b3db-ec153a3038b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:37:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:37:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:37:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:37:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:37:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:37:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:37:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:37:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:37:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:37:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:37:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:37:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:37:59,255 [237] [DEBUG] [app] Ending request: urn:request:3edea5b1-190e-41b8-96c3-13da8c371ecb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3edea5b1-190e-41b8-96c3-13da8c371ecb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:37:59,255 [235] [DEBUG] [app] Ending request: urn:request:1edef679-e8c8-4cf7-bb9b-33f8c6d55b21 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1edef679-e8c8-4cf7-bb9b-33f8c6d55b21', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:37:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:37:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:37:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:37:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:37:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:37:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:37:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:04.936688+00:00 (in 4.998337 seconds)
exportactionlogsworker stdout | 2025-02-07 05:37:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:59 UTC)" (scheduled at 2025-02-07 05:37:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:37:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:37:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:38:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:38:02,444 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:38:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:38:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:23.420886+00:00 (in 20.001319 seconds)
repositorygcworker stdout | 2025-02-07 05:38:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:03 UTC)" (scheduled at 2025-02-07 05:38:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:38:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:38:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 3, 419840), True, datetime.datetime(2025, 2, 7, 5, 38, 3, 419840), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:38:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:38:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:38:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:38:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:38:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:33.919286+00:00 (in 29.999516 seconds)
buildlogsarchiver stdout | 2025-02-07 05:38:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:33 UTC)" (scheduled at 2025-02-07 05:38:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:38:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 38, 3, 920057), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:38:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:38:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:38:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:38:04,118 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:38:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:38:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:59.937899+00:00 (in 55.000778 seconds)
exportactionlogsworker stdout | 2025-02-07 05:38:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:04 UTC)" (scheduled at 2025-02-07 05:38:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:38:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:38:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 4, 937407), True, datetime.datetime(2025, 2, 7, 5, 38, 4, 937407), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:38:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:38:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:38:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:38:05,299 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:38:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:38:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:18.732147+00:00 (in 9.999535 seconds)
notificationworker stdout | 2025-02-07 05:38:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:18 UTC)" (scheduled at 2025-02-07 05:38:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:38:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:38:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 8, 732901), True, datetime.datetime(2025, 2, 7, 5, 38, 8, 732901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:38:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:38:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:38:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:38:09,563 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:38:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:38:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:24.423055+00:00 (in 13.998363 seconds)
gcworker stdout | 2025-02-07 05:38:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:40 UTC)" (scheduled at 2025-02-07 05:38:10.424250+00:00)
gcworker stdout | 2025-02-07 05:38:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:38:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906390431, None, 1, 0])
gcworker stdout | 2025-02-07 05:38:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:38:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:38:10,614 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:38:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:38:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:41.831521+00:00 (in 29.999544 seconds)
autopruneworker stdout | 2025-02-07 05:38:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:41 UTC)" (scheduled at 2025-02-07 05:38:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:38:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903091836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:38:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:38:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:38:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:38:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:38:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:29.225378+00:00 (in 14.999480 seconds)
quotatotalworker stdout | 2025-02-07 05:38:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:29 UTC)" (scheduled at 2025-02-07 05:38:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:38:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:38:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:38:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:38:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:38:14,235 [238] [DEBUG] [app] Starting request: urn:request:f90d038a-3e71-4263-941e-a12bbdc67d71 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:14,235 [237] [DEBUG] [app] Starting request: urn:request:cd311379-fa37-46f5-9932-c7bc22bd8f93 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:38:14,238 [239] [DEBUG] [app] Starting request: urn:request:514bd42e-d0ac-4118-baec-e66ad5b0b52e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:14,238 [244] [DEBUG] [app] Starting request: urn:request:8a5b118f-00cc-405a-bc64-6b4cf65a9084 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:14,238 [239] [DEBUG] [app] Ending request: urn:request:514bd42e-d0ac-4118-baec-e66ad5b0b52e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:514bd42e-d0ac-4118-baec-e66ad5b0b52e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:14,238 [244] [DEBUG] [app] Ending request: urn:request:8a5b118f-00cc-405a-bc64-6b4cf65a9084 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8a5b118f-00cc-405a-bc64-6b4cf65a9084', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:38:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:38:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:14,241 [235] [DEBUG] [app] Starting request: urn:request:c8185a02-098d-4195-a010-3ae679da0341 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:14,242 [235] [DEBUG] [app] Ending request: urn:request:c8185a02-098d-4195-a010-3ae679da0341 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c8185a02-098d-4195-a010-3ae679da0341', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:38:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:14,242 [236] [DEBUG] [app] Starting request: urn:request:d31c541e-08d7-4141-a420-93efac8c8013 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:14,242 [236] [DEBUG] [app] Ending request: urn:request:d31c541e-08d7-4141-a420-93efac8c8013 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d31c541e-08d7-4141-a420-93efac8c8013', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:38:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:38:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:14,253 [238] [DEBUG] [app] Ending request: urn:request:f90d038a-3e71-4263-941e-a12bbdc67d71 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f90d038a-3e71-4263-941e-a12bbdc67d71', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:38:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:38:14,254 [237] [DEBUG] [app] Ending request: urn:request:cd311379-fa37-46f5-9932-c7bc22bd8f93 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd311379-fa37-46f5-9932-c7bc22bd8f93', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:38:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:38:14,614 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:38:15,157 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:38:15,767 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:38:17,202 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:38:17,902 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:38:17,905 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:38:17,910 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:38:17,912 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:38:17,915 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:38:17,921 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:38:17,989 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:38:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:38:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:28.732147+00:00 (in 9.999578 seconds)
notificationworker stdout | 2025-02-07 05:38:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:28 UTC)" (scheduled at 2025-02-07 05:38:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:38:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:38:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 18, 732838), True, datetime.datetime(2025, 2, 7, 5, 38, 18, 732838), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:38:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:38:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:38:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:38:18,919 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:38:18,922 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:38:18,924 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:38:19,294 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:19,505 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:38:19,971 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:38:19,990 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:38:20,299 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:38:20,365 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:38:20,374 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:38:20,377 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:38:20,379 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:38:20,388 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:38:20,392 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:38:20,394 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:38:20,397 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:38:20,401 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:38:20,694 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:38:21,013 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:38:21,311 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:38:21,387 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:38:23,034 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:38:23,080 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
repositorygcworker stdout | 2025-02-07 05:38:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:38:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:32.420707+00:00 (in 8.999359 seconds)
repositorygcworker stdout | 2025-02-07 05:38:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:23 UTC)" (scheduled at 2025-02-07 05:38:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:38:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:38:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:23 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:38:23,467 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:38:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:38:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:40.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 05:38:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:54 UTC)" (scheduled at 2025-02-07 05:38:24.423055+00:00)
gcworker stdout | 2025-02-07 05:38:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:38:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:38:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:38:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:38:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:38.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 05:38:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:38 UTC)" (scheduled at 2025-02-07 05:38:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:38:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:38:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 28, 732848), True, datetime.datetime(2025, 2, 7, 5, 38, 28, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:38:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:38:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:38:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:38:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:38:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:44.225378+00:00 (in 14.999507 seconds)
quotatotalworker stdout | 2025-02-07 05:38:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:44 UTC)" (scheduled at 2025-02-07 05:38:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:38:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:38:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:38:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:38:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:38:29,235 [237] [DEBUG] [app] Starting request: urn:request:43177610-0dd7-4793-a4f0-a56bcf22a1fa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:29,235 [238] [DEBUG] [app] Starting request: urn:request:83a4aaf4-b52d-450d-a595-8ca33b179659 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:38:29,238 [246] [DEBUG] [app] Starting request: urn:request:316e9a91-6bae-400f-8153-60c815e4b17d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:29,238 [239] [DEBUG] [app] Starting request: urn:request:60163c3d-ddda-41c2-a56f-99415c53f832 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:29,238 [239] [DEBUG] [app] Ending request: urn:request:60163c3d-ddda-41c2-a56f-99415c53f832 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:60163c3d-ddda-41c2-a56f-99415c53f832', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:29,239 [246] [DEBUG] [app] Ending request: urn:request:316e9a91-6bae-400f-8153-60c815e4b17d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:316e9a91-6bae-400f-8153-60c815e4b17d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:38:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:29,241 [238] [DEBUG] [app] Starting request: urn:request:c121d218-dc03-453b-be78-0b9c95068827 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:29,242 [238] [DEBUG] [app] Ending request: urn:request:c121d218-dc03-453b-be78-0b9c95068827 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c121d218-dc03-453b-be78-0b9c95068827', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:38:29,242 [236] [DEBUG] [app] Starting request: urn:request:9addcfcd-25fc-417e-b12a-42f62b7c3b36 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:38:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:29,242 [236] [DEBUG] [app] Ending request: urn:request:9addcfcd-25fc-417e-b12a-42f62b7c3b36 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9addcfcd-25fc-417e-b12a-42f62b7c3b36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:38:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:38:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:29,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:29,253 [238] [DEBUG] [app] Ending request: urn:request:83a4aaf4-b52d-450d-a595-8ca33b179659 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83a4aaf4-b52d-450d-a595-8ca33b179659', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:38:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:38:29,254 [237] [DEBUG] [app] Ending request: urn:request:43177610-0dd7-4793-a4f0-a56bcf22a1fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43177610-0dd7-4793-a4f0-a56bcf22a1fa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:38:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 05:38:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:38:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:32.625945+00:00 (in 1.004348 seconds)
namespacegcworker stdout | 2025-02-07 05:38:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:31 UTC)" (scheduled at 2025-02-07 05:38:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:38:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:38:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 31, 621915), True, datetime.datetime(2025, 2, 7, 5, 38, 31, 621915), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:38:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:38:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:38:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:38:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:38:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:03.419110+00:00 (in 30.997946 seconds)
repositorygcworker stdout | 2025-02-07 05:38:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:43:32 UTC)" (scheduled at 2025-02-07 05:38:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:38:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 38, 32, 421444), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:38:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 38, 32, 421444), True, datetime.datetime(2025, 2, 7, 5, 38, 32, 421444), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:38:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 38, 32, 421444), True, datetime.datetime(2025, 2, 7, 5, 38, 32, 421444), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 38, 32, 421444), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:38:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:38:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:43:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:38:32,481 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:38:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:38:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:31.621131+00:00 (in 58.994740 seconds)
namespacegcworker stdout | 2025-02-07 05:38:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:32 UTC)" (scheduled at 2025-02-07 05:38:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:38:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:38:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:38:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:38:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:03.919286+00:00 (in 29.999466 seconds)
buildlogsarchiver stdout | 2025-02-07 05:38:33,920 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:03 UTC)" (scheduled at 2025-02-07 05:38:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:38:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 38, 33, 920121), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:38:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:38:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:38:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:38:34,138 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:38:35,305 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:56.713547+00:00 (in 18.997210 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:37 UTC)" (scheduled at 2025-02-07 05:38:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:38:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:38:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:48.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 05:38:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:48 UTC)" (scheduled at 2025-02-07 05:38:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:38:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:38:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 38, 732862), True, datetime.datetime(2025, 2, 7, 5, 38, 38, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:38:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:38:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:38:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:38:39,599 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:38:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:38:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:54.423055+00:00 (in 13.998374 seconds)
gcworker stdout | 2025-02-07 05:38:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:10 UTC)" (scheduled at 2025-02-07 05:38:40.424250+00:00)
gcworker stdout | 2025-02-07 05:38:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:38:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906420430, None, 1, 0])
gcworker stdout | 2025-02-07 05:38:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:38:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:38:40,625 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:38:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:38:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:11.831521+00:00 (in 29.999556 seconds)
autopruneworker stdout | 2025-02-07 05:38:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:11 UTC)" (scheduled at 2025-02-07 05:38:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:38:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903121836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:38:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:38:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:38:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:38:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:38:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:59.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 05:38:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:59 UTC)" (scheduled at 2025-02-07 05:38:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:38:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:38:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:38:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:38:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:38:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:38:44,235 [235] [DEBUG] [app] Starting request: urn:request:fbcc0e4d-7463-4a24-aaee-7041d2b4751c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:44,235 [238] [DEBUG] [app] Starting request: urn:request:c3ed7653-f1b0-4a6b-881e-984312f483c3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:38:44,238 [239] [DEBUG] [app] Starting request: urn:request:8278eaa6-55d1-41cf-a21f-368e77fa32b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:44,238 [245] [DEBUG] [app] Starting request: urn:request:9afc662f-99c2-454f-ab9f-153414ec9983 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:44,238 [239] [DEBUG] [app] Ending request: urn:request:8278eaa6-55d1-41cf-a21f-368e77fa32b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8278eaa6-55d1-41cf-a21f-368e77fa32b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:44,238 [245] [DEBUG] [app] Ending request: urn:request:9afc662f-99c2-454f-ab9f-153414ec9983 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9afc662f-99c2-454f-ab9f-153414ec9983', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:38:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:44,241 [235] [DEBUG] [app] Starting request: urn:request:af394e7f-ff67-4cf5-a1a6-3c044432c34d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:44,241 [235] [DEBUG] [app] Ending request: urn:request:af394e7f-ff67-4cf5-a1a6-3c044432c34d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af394e7f-ff67-4cf5-a1a6-3c044432c34d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:38:44,241 [236] [DEBUG] [app] Starting request: urn:request:992d5665-c9e0-4c09-8dc4-7c012c30bbcc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:44,241 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:38:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:44,242 [236] [DEBUG] [app] Ending request: urn:request:992d5665-c9e0-4c09-8dc4-7c012c30bbcc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:992d5665-c9e0-4c09-8dc4-7c012c30bbcc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:38:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:38:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:44,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:44,252 [235] [DEBUG] [app] Ending request: urn:request:fbcc0e4d-7463-4a24-aaee-7041d2b4751c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fbcc0e4d-7463-4a24-aaee-7041d2b4751c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:38:44,253 [238] [DEBUG] [app] Ending request: urn:request:c3ed7653-f1b0-4a6b-881e-984312f483c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c3ed7653-f1b0-4a6b-881e-984312f483c3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:38:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:38:44,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
quotatotalworker stdout | 2025-02-07 05:38:44,650 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:38:45,193 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:38:45,802 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:38:47,230 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:38:47,909 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:38:47,913 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:38:47,917 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:38:47,920 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:38:47,923 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:38:47,947 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:38:48,025 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:38:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:38:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:50.814677+00:00 (in 2.082113 seconds)
notificationworker stdout | 2025-02-07 05:38:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:58 UTC)" (scheduled at 2025-02-07 05:38:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:38:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:38:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 48, 732834), True, datetime.datetime(2025, 2, 7, 5, 38, 48, 732834), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:38:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:38:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:38:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:38:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:38:48,927 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:38:48,931 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:38:48,933 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:38:49,328 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:49,519 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
globalpromstats stdout | 2025-02-07 05:38:50,007 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
servicekey stdout | 2025-02-07 05:38:50,010 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
exportactionlogsworker stdout | 2025-02-07 05:38:50,331 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:38:50,376 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:38:50,384 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:38:50,386 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:38:50,388 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:38:50,396 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:38:50,400 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:38:50,403 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:38:50,406 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:38:50,410 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:38:50,723 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:38:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:38:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:38:58.732147+00:00 (in 7.917000 seconds)
notificationworker stdout | 2025-02-07 05:38:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:50 UTC)" (scheduled at 2025-02-07 05:38:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:38:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:38:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:38:51,038 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:38:51,341 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:38:51,398 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:38:53,071 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:38:53,099 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:38:53,503 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:38:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:38:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:10.424250+00:00 (in 16.000737 seconds)
gcworker stdout | 2025-02-07 05:38:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:24 UTC)" (scheduled at 2025-02-07 05:38:54.423055+00:00)
gcworker stdout | 2025-02-07 05:38:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:38:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:38:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:38:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:56.118855+00:00 (in 59.999538 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:38:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:56 UTC)" (scheduled at 2025-02-07 05:38:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:38:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:38:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:38:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:37.715878+00:00 (in 41.001887 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:56 UTC)" (scheduled at 2025-02-07 05:38:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:38:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:38:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:38:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:08.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 05:38:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:08 UTC)" (scheduled at 2025-02-07 05:38:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:38:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:38:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 38, 58, 732839), True, datetime.datetime(2025, 2, 7, 5, 38, 58, 732839), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:38:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:38:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:38:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:38:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:38:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:14.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 05:38:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:14 UTC)" (scheduled at 2025-02-07 05:38:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:38:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:38:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:38:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:38:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:38:59,235 [238] [DEBUG] [app] Starting request: urn:request:7410885d-ff19-4eed-815a-6a29250d8b22 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:59,235 [237] [DEBUG] [app] Starting request: urn:request:a358ff41-20e6-4165-b997-5934ed64fa40 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:38:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:38:59,238 [239] [DEBUG] [app] Starting request: urn:request:de13117f-46e9-493b-8384-4a01483fed90 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:59,238 [245] [DEBUG] [app] Starting request: urn:request:7e7bb22d-def5-4378-9a4e-aecf4f68fbea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:38:59,238 [239] [DEBUG] [app] Ending request: urn:request:de13117f-46e9-493b-8384-4a01483fed90 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:de13117f-46e9-493b-8384-4a01483fed90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:59,238 [245] [DEBUG] [app] Ending request: urn:request:7e7bb22d-def5-4378-9a4e-aecf4f68fbea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7e7bb22d-def5-4378-9a4e-aecf4f68fbea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:38:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:38:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:38:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:38:59,241 [238] [DEBUG] [app] Starting request: urn:request:f5485c49-383c-467b-a173-a13be5a42fe3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:59,241 [235] [DEBUG] [app] Starting request: urn:request:3000e85c-a3fb-4076-8807-3d834d15417d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:38:59,241 [238] [DEBUG] [app] Ending request: urn:request:f5485c49-383c-467b-a173-a13be5a42fe3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f5485c49-383c-467b-a173-a13be5a42fe3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:38:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:59,242 [235] [DEBUG] [app] Ending request: urn:request:3000e85c-a3fb-4076-8807-3d834d15417d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3000e85c-a3fb-4076-8807-3d834d15417d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:38:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:38:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:38:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:38:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:38:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:38:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:59,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:38:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:38:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:38:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:38:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:38:59,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:59,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:38:59,253 [238] [DEBUG] [app] Ending request: urn:request:7410885d-ff19-4eed-815a-6a29250d8b22 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7410885d-ff19-4eed-815a-6a29250d8b22', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:38:59,253 [237] [DEBUG] [app] Ending request: urn:request:a358ff41-20e6-4165-b997-5934ed64fa40 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a358ff41-20e6-4165-b997-5934ed64fa40', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:38:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:38:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:38:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:38:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:38:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:38:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:38:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:04.936688+00:00 (in 4.998318 seconds)
exportactionlogsworker stdout | 2025-02-07 05:38:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:59 UTC)" (scheduled at 2025-02-07 05:38:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:38:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:38:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:39:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:39:02,517 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:39:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:39:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:23.420886+00:00 (in 20.001297 seconds)
repositorygcworker stdout | 2025-02-07 05:39:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:03 UTC)" (scheduled at 2025-02-07 05:39:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:39:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:39:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 3, 419869), True, datetime.datetime(2025, 2, 7, 5, 39, 3, 419869), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:39:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:39:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:39:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:39:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:39:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:33.919286+00:00 (in 29.999498 seconds)
buildlogsarchiver stdout | 2025-02-07 05:39:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:33 UTC)" (scheduled at 2025-02-07 05:39:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:39:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 39, 3, 920051), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:39:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:39:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:39:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:39:04,172 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:39:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:39:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:52.937648+00:00 (in 48.000496 seconds)
exportactionlogsworker stdout | 2025-02-07 05:39:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:04 UTC)" (scheduled at 2025-02-07 05:39:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:39:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:39:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 4, 937509), True, datetime.datetime(2025, 2, 7, 5, 39, 4, 937509), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:39:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:39:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:39:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:39:05,339 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:39:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:39:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:18.732147+00:00 (in 9.999573 seconds)
notificationworker stdout | 2025-02-07 05:39:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:18 UTC)" (scheduled at 2025-02-07 05:39:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:39:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:39:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 8, 732845), True, datetime.datetime(2025, 2, 7, 5, 39, 8, 732845), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:39:08,743 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:39:08,743 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:39:08,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:39:09,635 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:39:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:39:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:24.423055+00:00 (in 13.998345 seconds)
gcworker stdout | 2025-02-07 05:39:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:40 UTC)" (scheduled at 2025-02-07 05:39:10.424250+00:00)
gcworker stdout | 2025-02-07 05:39:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:39:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906450431, None, 1, 0])
gcworker stdout | 2025-02-07 05:39:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:39:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:39:10,641 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:39:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:39:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:41.831521+00:00 (in 29.999558 seconds)
autopruneworker stdout | 2025-02-07 05:39:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:41 UTC)" (scheduled at 2025-02-07 05:39:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:39:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903151836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:39:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:39:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:39:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:39:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:39:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:29.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 05:39:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:29 UTC)" (scheduled at 2025-02-07 05:39:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:39:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:39:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:39:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:39:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:39:14,235 [235] [DEBUG] [app] Starting request: urn:request:8a5714cf-6cff-4c79-8cb6-b50a35104959 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:14,235 [237] [DEBUG] [app] Starting request: urn:request:8dcea685-1313-48e9-b1a8-e3d0b4257194 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:39:14,238 [246] [DEBUG] [app] Starting request: urn:request:8444caf3-6d39-41a6-8950-22bbe262f3b7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:14,238 [239] [DEBUG] [app] Starting request: urn:request:dca8a8be-9bfa-45dc-8db3-3cc5e6e2cf8b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:14,239 [239] [DEBUG] [app] Ending request: urn:request:dca8a8be-9bfa-45dc-8db3-3cc5e6e2cf8b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dca8a8be-9bfa-45dc-8db3-3cc5e6e2cf8b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:39:14,239 [246] [DEBUG] [app] Ending request: urn:request:8444caf3-6d39-41a6-8950-22bbe262f3b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8444caf3-6d39-41a6-8950-22bbe262f3b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:39:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:39:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:39:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:14,241 [236] [DEBUG] [app] Starting request: urn:request:dcf096d8-9536-46d5-b403-6fa5877edf73 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:14,242 [236] [DEBUG] [app] Ending request: urn:request:dcf096d8-9536-46d5-b403-6fa5877edf73 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dcf096d8-9536-46d5-b403-6fa5877edf73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:14,242 [237] [DEBUG] [app] Starting request: urn:request:cde53837-2ed3-40aa-9723-a8b4617eaa27 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:39:14,242 [237] [DEBUG] [app] Ending request: urn:request:cde53837-2ed3-40aa-9723-a8b4617eaa27 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cde53837-2ed3-40aa-9723-a8b4617eaa27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:39:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:14,254 [237] [DEBUG] [app] Ending request: urn:request:8dcea685-1313-48e9-b1a8-e3d0b4257194 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8dcea685-1313-48e9-b1a8-e3d0b4257194', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:14,254 [235] [DEBUG] [app] Ending request: urn:request:8a5714cf-6cff-4c79-8cb6-b50a35104959 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a5714cf-6cff-4c79-8cb6-b50a35104959', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:39:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:39:14,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:39:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:39:14,682 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:39:15,230 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:39:15,831 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:39:17,267 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:39:17,918 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:39:17,920 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:39:17,924 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:39:17,926 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:39:17,930 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:39:17,957 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:39:18,035 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:39:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:39:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:28.732147+00:00 (in 9.999580 seconds)
notificationworker stdout | 2025-02-07 05:39:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:28 UTC)" (scheduled at 2025-02-07 05:39:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:39:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:39:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 18, 732835), True, datetime.datetime(2025, 2, 7, 5, 39, 18, 732835), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:39:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:39:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:39:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:39:18,936 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:39:18,939 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:39:18,942 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:39:19,359 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:19,551 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:39:20,017 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:39:20,041 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:39:20,359 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:39:20,386 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:39:20,394 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:39:20,397 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:39:20,399 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:39:20,403 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:39:20,407 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:39:20,410 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:39:20,414 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:39:20,418 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:39:20,747 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:39:21,065 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:39:21,378 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:39:21,417 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:39:23,107 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:39:23,134 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
repositorygcworker stdout | 2025-02-07 05:39:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:39:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:03.419110+00:00 (in 39.997755 seconds)
repositorygcworker stdout | 2025-02-07 05:39:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:23 UTC)" (scheduled at 2025-02-07 05:39:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:39:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:39:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:23 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:39:23,539 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:39:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:39:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:40.424250+00:00 (in 16.000740 seconds)
gcworker stdout | 2025-02-07 05:39:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:54 UTC)" (scheduled at 2025-02-07 05:39:24.423055+00:00)
gcworker stdout | 2025-02-07 05:39:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:39:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:39:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:39:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:39:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:38.732147+00:00 (in 9.999571 seconds)
notificationworker stdout | 2025-02-07 05:39:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:38 UTC)" (scheduled at 2025-02-07 05:39:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:39:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:39:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 28, 732875), True, datetime.datetime(2025, 2, 7, 5, 39, 28, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:39:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:39:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:39:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:39:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:39:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:44.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 05:39:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:44 UTC)" (scheduled at 2025-02-07 05:39:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:39:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:39:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:39:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:39:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:39:29,236 [235] [DEBUG] [app] Starting request: urn:request:2aeeec07-11df-403a-aea1-2902dced9625 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:29,236 [238] [DEBUG] [app] Starting request: urn:request:5db7d153-6beb-4ad9-89c6-2e7d14f43a12 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:29,238 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:39:29,239 [246] [DEBUG] [app] Starting request: urn:request:546a7d81-fa70-4461-b362-eac1b3094795 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:29,239 [239] [DEBUG] [app] Starting request: urn:request:68928637-b31d-4a34-aa19-6b592f907363 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:29,240 [246] [DEBUG] [app] Ending request: urn:request:546a7d81-fa70-4461-b362-eac1b3094795 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:546a7d81-fa70-4461-b362-eac1b3094795', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:39:29,240 [239] [DEBUG] [app] Ending request: urn:request:68928637-b31d-4a34-aa19-6b592f907363 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:68928637-b31d-4a34-aa19-6b592f907363', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:39:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:39:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:39:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:29,242 [238] [DEBUG] [app] Starting request: urn:request:0981f338-2f48-421b-85e2-770a35813d26 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:29,243 [238] [DEBUG] [app] Ending request: urn:request:0981f338-2f48-421b-85e2-770a35813d26 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0981f338-2f48-421b-85e2-770a35813d26', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:29,243 [237] [DEBUG] [app] Starting request: urn:request:cccd5f4d-c678-46bc-ad56-95e1e9ec4e7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:39:29,243 [237] [DEBUG] [app] Ending request: urn:request:cccd5f4d-c678-46bc-ad56-95e1e9ec4e7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cccd5f4d-c678-46bc-ad56-95e1e9ec4e7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:39:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:29,254 [238] [DEBUG] [app] Ending request: urn:request:5db7d153-6beb-4ad9-89c6-2e7d14f43a12 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5db7d153-6beb-4ad9-89c6-2e7d14f43a12', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:39:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:39:29,255 [235] [DEBUG] [app] Ending request: urn:request:2aeeec07-11df-403a-aea1-2902dced9625 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2aeeec07-11df-403a-aea1-2902dced9625', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:39:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:39:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:39:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:32.625945+00:00 (in 1.004350 seconds)
namespacegcworker stdout | 2025-02-07 05:39:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:31 UTC)" (scheduled at 2025-02-07 05:39:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:39:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:39:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 31, 621876), True, datetime.datetime(2025, 2, 7, 5, 39, 31, 621876), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:39:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:39:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:39:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:39:32,535 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:39:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:39:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:31.621131+00:00 (in 58.994733 seconds)
namespacegcworker stdout | 2025-02-07 05:39:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:32 UTC)" (scheduled at 2025-02-07 05:39:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:39:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:39:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:39:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:39:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:03.919286+00:00 (in 29.999573 seconds)
buildlogsarchiver stdout | 2025-02-07 05:39:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:03 UTC)" (scheduled at 2025-02-07 05:39:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:39:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 39, 33, 919971), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:39:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:39:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:39:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:39:34,208 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:39:35,369 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:56.713547+00:00 (in 18.997178 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:37 UTC)" (scheduled at 2025-02-07 05:39:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:39:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:39:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:48.732147+00:00 (in 9.999581 seconds)
notificationworker stdout | 2025-02-07 05:39:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:48 UTC)" (scheduled at 2025-02-07 05:39:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:39:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:39:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 38, 732830), True, datetime.datetime(2025, 2, 7, 5, 39, 38, 732830), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:39:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:39:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:39:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:39:39,645 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:39:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:39:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:54.423055+00:00 (in 13.998340 seconds)
gcworker stdout | 2025-02-07 05:39:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:10 UTC)" (scheduled at 2025-02-07 05:39:40.424250+00:00)
gcworker stdout | 2025-02-07 05:39:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:39:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906480431, None, 1, 0])
gcworker stdout | 2025-02-07 05:39:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:39:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:39:40,677 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:39:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:39:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:11.831521+00:00 (in 29.999563 seconds)
autopruneworker stdout | 2025-02-07 05:39:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:11 UTC)" (scheduled at 2025-02-07 05:39:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:39:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903181836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:39:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:39:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:39:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:39:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:39:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:59.225378+00:00 (in 14.999556 seconds)
quotatotalworker stdout | 2025-02-07 05:39:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:59 UTC)" (scheduled at 2025-02-07 05:39:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:39:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:39:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:39:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:39:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:39:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:39:44,236 [235] [DEBUG] [app] Starting request: urn:request:39125310-3d11-4fc5-9ead-2bb53bdfa5a7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:44,236 [238] [DEBUG] [app] Starting request: urn:request:788117a8-f79a-4b74-a539-2e4cfaff9c00 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:39:44,239 [239] [DEBUG] [app] Starting request: urn:request:2d2ea89f-5989-4de2-bc3e-6e7ed5fd4ffb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:44,239 [246] [DEBUG] [app] Starting request: urn:request:566eab2c-b5ed-4d8a-bae4-1f08e841792d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:44,239 [239] [DEBUG] [app] Ending request: urn:request:2d2ea89f-5989-4de2-bc3e-6e7ed5fd4ffb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d2ea89f-5989-4de2-bc3e-6e7ed5fd4ffb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:39:44,239 [246] [DEBUG] [app] Ending request: urn:request:566eab2c-b5ed-4d8a-bae4-1f08e841792d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:566eab2c-b5ed-4d8a-bae4-1f08e841792d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:39:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:39:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:39:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:39:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:44,242 [236] [DEBUG] [app] Starting request: urn:request:72573c27-5951-46b1-a0ae-9abeff85403e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:44,242 [236] [DEBUG] [app] Ending request: urn:request:72573c27-5951-46b1-a0ae-9abeff85403e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:72573c27-5951-46b1-a0ae-9abeff85403e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:39:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:44,243 [237] [DEBUG] [app] Starting request: urn:request:fd5cc37b-54ff-4820-9122-b73b226df0bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:44,243 [237] [DEBUG] [app] Ending request: urn:request:fd5cc37b-54ff-4820-9122-b73b226df0bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fd5cc37b-54ff-4820-9122-b73b226df0bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:39:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:39:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:44,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:44,255 [235] [DEBUG] [app] Ending request: urn:request:39125310-3d11-4fc5-9ead-2bb53bdfa5a7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39125310-3d11-4fc5-9ead-2bb53bdfa5a7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:44,255 [238] [DEBUG] [app] Ending request: urn:request:788117a8-f79a-4b74-a539-2e4cfaff9c00 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:788117a8-f79a-4b74-a539-2e4cfaff9c00', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:39:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 05:39:44,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:39:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
quotatotalworker stdout | 2025-02-07 05:39:44,717 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:39:45,267 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:39:45,867 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:39:47,274 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:39:47,926 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:39:47,930 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:39:47,932 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:39:47,935 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:39:47,938 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:39:47,975 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:39:48,068 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:39:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:39:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:50.814677+00:00 (in 2.082121 seconds)
notificationworker stdout | 2025-02-07 05:39:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:58 UTC)" (scheduled at 2025-02-07 05:39:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:39:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:39:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 48, 732825), True, datetime.datetime(2025, 2, 7, 5, 39, 48, 732825), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:39:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:39:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:39:48,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:39:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:39:48,943 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:39:48,946 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:39:48,949 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:39:49,395 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:49,587 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:39:50,032 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:39:50,077 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:39:50,394 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:39:50,397 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:39:50,403 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:39:50,406 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:39:50,409 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:39:50,411 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:39:50,417 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:39:50,419 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:39:50,425 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
gunicorn-registry stdout | 2025-02-07 05:39:50,427 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
repositorygcworker stdout | 2025-02-07 05:39:50,774 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:39:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:39:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:58.732147+00:00 (in 7.917038 seconds)
notificationworker stdout | 2025-02-07 05:39:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:50 UTC)" (scheduled at 2025-02-07 05:39:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:39:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:39:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:39:51,101 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:39:51,414 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:39:51,453 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
exportactionlogsworker stdout | 2025-02-07 05:39:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:39:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:39:59.937899+00:00 (in 6.999755 seconds)
exportactionlogsworker stdout | 2025-02-07 05:39:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:44:52 UTC)" (scheduled at 2025-02-07 05:39:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:39:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 39, 52, 938440), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:39:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 39, 52, 938440), True, datetime.datetime(2025, 2, 7, 5, 39, 52, 938440), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:39:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 39, 52, 938440), True, datetime.datetime(2025, 2, 7, 5, 39, 52, 938440), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 39, 52, 938440), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:39:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:39:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:44:52 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:39:53,118 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:39:53,161 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:39:53,556 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:39:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:39:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:10.424250+00:00 (in 16.000746 seconds)
gcworker stdout | 2025-02-07 05:39:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:24 UTC)" (scheduled at 2025-02-07 05:39:54.423055+00:00)
gcworker stdout | 2025-02-07 05:39:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:39:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:39:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:39:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:56.118855+00:00 (in 59.999510 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:39:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:56 UTC)" (scheduled at 2025-02-07 05:39:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:39:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:39:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:39:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:37.715878+00:00 (in 41.001838 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:56 UTC)" (scheduled at 2025-02-07 05:39:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:39:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:39:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:39:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:08.732147+00:00 (in 9.999589 seconds)
notificationworker stdout | 2025-02-07 05:39:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:08 UTC)" (scheduled at 2025-02-07 05:39:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:39:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:39:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 39, 58, 732825), True, datetime.datetime(2025, 2, 7, 5, 39, 58, 732825), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:39:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:39:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:39:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:39:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:39:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:14.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 05:39:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:14 UTC)" (scheduled at 2025-02-07 05:39:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:39:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:39:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:39:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:39:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:39:59,236 [235] [DEBUG] [app] Starting request: urn:request:6a3edb69-1ba2-4440-b9c2-9aa399fb45bd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:59,236 [238] [DEBUG] [app] Starting request: urn:request:8f0cdbae-7576-46c9-b350-34a5dd4e3b1e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:39:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:39:59,239 [246] [DEBUG] [app] Starting request: urn:request:1c0498ed-4ac7-4020-bb2d-3d1d068fd2af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:59,239 [239] [DEBUG] [app] Starting request: urn:request:dcf9b3dd-6bf4-44c7-bd4f-9b16363cc3ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:39:59,240 [239] [DEBUG] [app] Ending request: urn:request:dcf9b3dd-6bf4-44c7-bd4f-9b16363cc3ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dcf9b3dd-6bf4-44c7-bd4f-9b16363cc3ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:39:59,240 [246] [DEBUG] [app] Ending request: urn:request:1c0498ed-4ac7-4020-bb2d-3d1d068fd2af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c0498ed-4ac7-4020-bb2d-3d1d068fd2af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:39:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:39:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:39:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:39:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:59,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:39:59,243 [237] [DEBUG] [app] Starting request: urn:request:28f16e22-7b06-4c06-acfd-9361446878f4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:59,243 [236] [DEBUG] [app] Starting request: urn:request:5a538d04-5b34-40cd-9737-99365703a0fb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:39:59,244 [236] [DEBUG] [app] Ending request: urn:request:5a538d04-5b34-40cd-9737-99365703a0fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5a538d04-5b34-40cd-9737-99365703a0fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:59,244 [237] [DEBUG] [app] Ending request: urn:request:28f16e22-7b06-4c06-acfd-9361446878f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28f16e22-7b06-4c06-acfd-9361446878f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:39:59,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:39:59,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:39:59,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:39:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:39:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:39:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:39:59,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:59,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:39:59,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:59,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:39:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:59,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:59,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:39:59,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:39:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:59,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:39:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:39:59,255 [238] [DEBUG] [app] Ending request: urn:request:8f0cdbae-7576-46c9-b350-34a5dd4e3b1e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8f0cdbae-7576-46c9-b350-34a5dd4e3b1e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:59,255 [235] [DEBUG] [app] Ending request: urn:request:6a3edb69-1ba2-4440-b9c2-9aa399fb45bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a3edb69-1ba2-4440-b9c2-9aa399fb45bd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:39:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:39:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:39:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:39:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:39:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:39:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
exportactionlogsworker stdout | 2025-02-07 05:39:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:39:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:04.936688+00:00 (in 4.998237 seconds)
exportactionlogsworker stdout | 2025-02-07 05:39:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:59 UTC)" (scheduled at 2025-02-07 05:39:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:39:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:39:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:40:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:40:02,571 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:40:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:40:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:23.420886+00:00 (in 20.001281 seconds)
repositorygcworker stdout | 2025-02-07 05:40:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:03 UTC)" (scheduled at 2025-02-07 05:40:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:40:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:40:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 3, 419890), True, datetime.datetime(2025, 2, 7, 5, 40, 3, 419890), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:40:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:40:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:40:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:40:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:40:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:33.919286+00:00 (in 29.999547 seconds)
buildlogsarchiver stdout | 2025-02-07 05:40:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:33 UTC)" (scheduled at 2025-02-07 05:40:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:40:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 40, 3, 919987), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:40:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:40:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:40:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:40:04,230 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:40:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:40:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:59.937899+00:00 (in 55.000659 seconds)
exportactionlogsworker stdout | 2025-02-07 05:40:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:04 UTC)" (scheduled at 2025-02-07 05:40:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:40:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:40:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 4, 937569), True, datetime.datetime(2025, 2, 7, 5, 40, 4, 937569), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:40:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:40:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:40:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:40:05,403 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:40:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:40:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:18.732147+00:00 (in 9.999526 seconds)
notificationworker stdout | 2025-02-07 05:40:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:18 UTC)" (scheduled at 2025-02-07 05:40:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:40:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:40:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 8, 732935), True, datetime.datetime(2025, 2, 7, 5, 40, 8, 732935), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:40:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:40:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:40:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:40:09,674 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:40:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:40:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:24.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 05:40:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:40 UTC)" (scheduled at 2025-02-07 05:40:10.424250+00:00)
gcworker stdout | 2025-02-07 05:40:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:40:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906510431, None, 1, 0])
gcworker stdout | 2025-02-07 05:40:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:40:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:40:10,713 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:40:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:40:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:41.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 05:40:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:41 UTC)" (scheduled at 2025-02-07 05:40:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:40:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903211836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:40:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:40:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:40:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:40:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:40:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:29.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 05:40:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:29 UTC)" (scheduled at 2025-02-07 05:40:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:40:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:40:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:40:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:40:14,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:40:14,235 [235] [DEBUG] [app] Starting request: urn:request:5d448875-6e3c-42d1-aef3-bd695af6785f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:14,235 [238] [DEBUG] [app] Starting request: urn:request:0afd21fd-b13d-40ae-9592-53ba89f7a6cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:40:14,238 [245] [DEBUG] [app] Starting request: urn:request:53e41a9e-1ba8-49bb-b9dd-0f947d5086e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:40:14,238 [239] [DEBUG] [app] Starting request: urn:request:96bcdc67-40ef-4b37-835c-c49c7fa71cff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:40:14,238 [245] [DEBUG] [app] Ending request: urn:request:53e41a9e-1ba8-49bb-b9dd-0f947d5086e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:53e41a9e-1ba8-49bb-b9dd-0f947d5086e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:40:14,239 [239] [DEBUG] [app] Ending request: urn:request:96bcdc67-40ef-4b37-835c-c49c7fa71cff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96bcdc67-40ef-4b37-835c-c49c7fa71cff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:40:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:40:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:14,241 [237] [DEBUG] [app] Starting request: urn:request:ce6d75ea-1126-48c6-84d1-1f016bb8f6fb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:40:14,242 [237] [DEBUG] [app] Ending request: urn:request:ce6d75ea-1126-48c6-84d1-1f016bb8f6fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ce6d75ea-1126-48c6-84d1-1f016bb8f6fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:40:14,242 [238] [DEBUG] [app] Starting request: urn:request:eb0b61f4-a650-4400-abc2-1174f1821107 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:40:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:14,242 [238] [DEBUG] [app] Ending request: urn:request:eb0b61f4-a650-4400-abc2-1174f1821107 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb0b61f4-a650-4400-abc2-1174f1821107', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:40:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:40:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:40:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:14,254 [235] [DEBUG] [app] Ending request: urn:request:5d448875-6e3c-42d1-aef3-bd695af6785f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d448875-6e3c-42d1-aef3-bd695af6785f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:14,254 [238] [DEBUG] [app] Ending request: urn:request:0afd21fd-b13d-40ae-9592-53ba89f7a6cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0afd21fd-b13d-40ae-9592-53ba89f7a6cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:40:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:40:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:40:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:40:14,733 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:40:15,302 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:40:15,901 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:40:17,287 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:40:17,935 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:40:17,938 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:40:17,941 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:40:17,943 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:40:17,947 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:40:18,012 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:40:18,105 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:40:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:40:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:28.732147+00:00 (in 9.999480 seconds)
notificationworker stdout | 2025-02-07 05:40:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:28 UTC)" (scheduled at 2025-02-07 05:40:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:40:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:40:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 18, 732994), True, datetime.datetime(2025, 2, 7, 5, 40, 18, 732994), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:40:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:40:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:40:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:40:18,951 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:40:18,954 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:40:18,958 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:40:19,421 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:19,618 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:40:20,068 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:40:20,113 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:40:20,406 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:40:20,411 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:40:20,414 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:40:20,417 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:40:20,422 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:40:20,427 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
exportactionlogsworker stdout | 2025-02-07 05:40:20,430 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
gunicorn-registry stdout | 2025-02-07 05:40:20,434 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:40:20,437 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:40:20,439 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
repositorygcworker stdout | 2025-02-07 05:40:20,793 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:40:21,137 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:40:21,450 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:40:21,490 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:40:23,154 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:40:23,197 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
repositorygcworker stdout | 2025-02-07 05:40:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:40:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:03.419110+00:00 (in 39.997745 seconds)
repositorygcworker stdout | 2025-02-07 05:40:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:23 UTC)" (scheduled at 2025-02-07 05:40:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:40:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:40:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:23 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:40:23,579 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:40:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:40:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:40.424250+00:00 (in 16.000726 seconds)
gcworker stdout | 2025-02-07 05:40:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:54 UTC)" (scheduled at 2025-02-07 05:40:24.423055+00:00)
gcworker stdout | 2025-02-07 05:40:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:40:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:40:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:40:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:40:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:38.732147+00:00 (in 9.999483 seconds)
notificationworker stdout | 2025-02-07 05:40:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:38 UTC)" (scheduled at 2025-02-07 05:40:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:40:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:40:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 28, 732973), True, datetime.datetime(2025, 2, 7, 5, 40, 28, 732973), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:40:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:40:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:40:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:40:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:40:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:44.225378+00:00 (in 14.999558 seconds)
quotatotalworker stdout | 2025-02-07 05:40:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:44 UTC)" (scheduled at 2025-02-07 05:40:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:40:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:40:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:40:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:40:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:40:29,235 [238] [DEBUG] [app] Starting request: urn:request:438cad5f-0f20-476d-a20b-13aef3bcb0b0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:29,235 [237] [DEBUG] [app] Starting request: urn:request:b9a3dcc3-2fdf-4dc6-b849-e3c742d7181a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:40:29,238 [239] [DEBUG] [app] Starting request: urn:request:724672dc-60ec-4ac4-9959-057626ca8035 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:40:29,238 [246] [DEBUG] [app] Starting request: urn:request:76881798-1362-40bf-8a82-1236919b78b5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:40:29,238 [239] [DEBUG] [app] Ending request: urn:request:724672dc-60ec-4ac4-9959-057626ca8035 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:724672dc-60ec-4ac4-9959-057626ca8035', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:40:29,238 [246] [DEBUG] [app] Ending request: urn:request:76881798-1362-40bf-8a82-1236919b78b5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76881798-1362-40bf-8a82-1236919b78b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:40:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:40:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:29,241 [237] [DEBUG] [app] Starting request: urn:request:cb73a553-f089-44d3-911e-9d0e22bdaee0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:40:29,241 [236] [DEBUG] [app] Starting request: urn:request:2036b908-ff0d-46dc-906c-a1e657d6876f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:40:29,241 [237] [DEBUG] [app] Ending request: urn:request:cb73a553-f089-44d3-911e-9d0e22bdaee0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cb73a553-f089-44d3-911e-9d0e22bdaee0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:40:29,241 [236] [DEBUG] [app] Ending request: urn:request:2036b908-ff0d-46dc-906c-a1e657d6876f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2036b908-ff0d-46dc-906c-a1e657d6876f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:40:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:40:29,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:40:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:40:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:29,253 [238] [DEBUG] [app] Ending request: urn:request:438cad5f-0f20-476d-a20b-13aef3bcb0b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:438cad5f-0f20-476d-a20b-13aef3bcb0b0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:29,253 [237] [DEBUG] [app] Ending request: urn:request:b9a3dcc3-2fdf-4dc6-b849-e3c742d7181a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b9a3dcc3-2fdf-4dc6-b849-e3c742d7181a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:40:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:40:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:40:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 05:40:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:40:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:32.625945+00:00 (in 1.004342 seconds)
namespacegcworker stdout | 2025-02-07 05:40:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:31 UTC)" (scheduled at 2025-02-07 05:40:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:40:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:40:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 31, 621878), True, datetime.datetime(2025, 2, 7, 5, 40, 31, 621878), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:40:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:40:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:40:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:31 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:40:32,607 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
namespacegcworker stdout | 2025-02-07 05:40:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:40:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:31.621131+00:00 (in 58.994716 seconds)
namespacegcworker stdout | 2025-02-07 05:40:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:32 UTC)" (scheduled at 2025-02-07 05:40:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:40:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:40:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:32 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:40:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:40:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:03.919286+00:00 (in 29.999556 seconds)
buildlogsarchiver stdout | 2025-02-07 05:40:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:03 UTC)" (scheduled at 2025-02-07 05:40:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:40:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 40, 33, 919972), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:40:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:40:33,926 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:40:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:40:34,266 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:40:35,432 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:56.713547+00:00 (in 18.997184 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:37 UTC)" (scheduled at 2025-02-07 05:40:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:40:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:40:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:48.732147+00:00 (in 9.999524 seconds)
notificationworker stdout | 2025-02-07 05:40:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:48 UTC)" (scheduled at 2025-02-07 05:40:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:40:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:40:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 38, 732903), True, datetime.datetime(2025, 2, 7, 5, 40, 38, 732903), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:40:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:40:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:40:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:40:39,707 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:40:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:40:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:54.423055+00:00 (in 13.998357 seconds)
gcworker stdout | 2025-02-07 05:40:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:10 UTC)" (scheduled at 2025-02-07 05:40:40.424250+00:00)
gcworker stdout | 2025-02-07 05:40:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:40:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906540431, None, 1, 0])
gcworker stdout | 2025-02-07 05:40:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:40:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:40:40,749 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:40:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:40:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:11.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 05:40:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:11 UTC)" (scheduled at 2025-02-07 05:40:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:40:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903241836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:40:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:40:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:40:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:40:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:40:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:59.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 05:40:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:59 UTC)" (scheduled at 2025-02-07 05:40:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:40:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:40:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:40:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:40:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:40:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:40:44,235 [235] [DEBUG] [app] Starting request: urn:request:3055f42b-18c8-486f-87f6-c8ab94e7d887 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:44,235 [238] [DEBUG] [app] Starting request: urn:request:f1810501-09cd-4233-bf8f-a74f0011fb65 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:40:44,238 [239] [DEBUG] [app] Starting request: urn:request:08b78e63-0b8b-4345-8e00-702ed998e642 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:40:44,238 [244] [DEBUG] [app] Starting request: urn:request:96524c4d-3e99-4072-bd0e-7991d6ba71bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:40:44,239 [239] [DEBUG] [app] Ending request: urn:request:08b78e63-0b8b-4345-8e00-702ed998e642 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08b78e63-0b8b-4345-8e00-702ed998e642', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:40:44,239 [244] [DEBUG] [app] Ending request: urn:request:96524c4d-3e99-4072-bd0e-7991d6ba71bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96524c4d-3e99-4072-bd0e-7991d6ba71bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:40:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:40:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:40:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:44,241 [238] [DEBUG] [app] Starting request: urn:request:c4c5e194-8dd4-461d-a2f9-b6308e565699 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:40:44,241 [237] [DEBUG] [app] Starting request: urn:request:3f06a6d2-53be-404e-90f4-241a8d0ca918 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:40:44,242 [238] [DEBUG] [app] Ending request: urn:request:c4c5e194-8dd4-461d-a2f9-b6308e565699 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c4c5e194-8dd4-461d-a2f9-b6308e565699', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:40:44,242 [237] [DEBUG] [app] Ending request: urn:request:3f06a6d2-53be-404e-90f4-241a8d0ca918 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f06a6d2-53be-404e-90f4-241a8d0ca918', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:40:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:44,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:40:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:44,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:44,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:40:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:40:44,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:44,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:44,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:44,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:44,253 [235] [DEBUG] [app] Ending request: urn:request:3055f42b-18c8-486f-87f6-c8ab94e7d887 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3055f42b-18c8-486f-87f6-c8ab94e7d887', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:40:44,253 [238] [DEBUG] [app] Ending request: urn:request:f1810501-09cd-4233-bf8f-a74f0011fb65 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1810501-09cd-4233-bf8f-a74f0011fb65', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:40:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 05:40:44,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
quotatotalworker stdout | 2025-02-07 05:40:44,747 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:40:45,330 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:40:45,937 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:40:47,314 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:40:47,944 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:40:47,947 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:40:47,950 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:40:47,953 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:40:47,957 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:40:48,041 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:40:48,120 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:40:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:40:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:50.814677+00:00 (in 2.082113 seconds)
notificationworker stdout | 2025-02-07 05:40:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:58 UTC)" (scheduled at 2025-02-07 05:40:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:40:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:40:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 48, 732843), True, datetime.datetime(2025, 2, 7, 5, 40, 48, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:40:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:40:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:40:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:40:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:40:48,960 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:40:48,962 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:40:48,965 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:40:49,445 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:49,649 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:40:50,082 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:40:50,148 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:40:50,414 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:40:50,418 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:40:50,421 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:40:50,428 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:40:50,432 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:40:50,437 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:40:50,443 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:40:50,446 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:40:50,449 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:40:50,465 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:40:50,811 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
notificationworker stdout | 2025-02-07 05:40:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:40:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:40:58.732147+00:00 (in 7.917057 seconds)
notificationworker stdout | 2025-02-07 05:40:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:50 UTC)" (scheduled at 2025-02-07 05:40:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:40:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:40:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:50 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:40:51,173 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:40:51,479 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:40:51,525 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:40:53,178 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:40:53,229 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:40:53,615 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:40:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:40:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:10.424250+00:00 (in 16.000698 seconds)
gcworker stdout | 2025-02-07 05:40:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:24 UTC)" (scheduled at 2025-02-07 05:40:54.423055+00:00)
gcworker stdout | 2025-02-07 05:40:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:40:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:40:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:40:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:56.118855+00:00 (in 59.999493 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:40:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:56 UTC)" (scheduled at 2025-02-07 05:40:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:40:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:40:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:40:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:37.715878+00:00 (in 41.001889 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:56 UTC)" (scheduled at 2025-02-07 05:40:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:40:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:40:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:40:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:08.732147+00:00 (in 9.999586 seconds)
notificationworker stdout | 2025-02-07 05:40:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:08 UTC)" (scheduled at 2025-02-07 05:40:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:40:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:40:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 40, 58, 732823), True, datetime.datetime(2025, 2, 7, 5, 40, 58, 732823), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:40:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:40:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:40:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:40:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:40:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:14.225378+00:00 (in 14.999549 seconds)
quotatotalworker stdout | 2025-02-07 05:40:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:14 UTC)" (scheduled at 2025-02-07 05:40:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:40:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:40:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:40:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:40:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:40:59,235 [235] [DEBUG] [app] Starting request: urn:request:469c4c89-0af7-4ae3-b777-06f3cf1f457d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:59,235 [237] [DEBUG] [app] Starting request: urn:request:a6c24168-2ad7-4c7e-b8e0-2b29068a25e7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:40:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:40:59,238 [239] [DEBUG] [app] Starting request: urn:request:a999c3c2-13c9-42c0-ab44-6c39a02a661e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:40:59,238 [239] [DEBUG] [app] Ending request: urn:request:a999c3c2-13c9-42c0-ab44-6c39a02a661e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a999c3c2-13c9-42c0-ab44-6c39a02a661e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:40:59,238 [244] [DEBUG] [app] Starting request: urn:request:2159129b-b386-4a99-a088-4e2c085c408a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:40:59,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:40:59,239 [244] [DEBUG] [app] Ending request: urn:request:2159129b-b386-4a99-a088-4e2c085c408a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2159129b-b386-4a99-a088-4e2c085c408a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:40:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:40:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:40:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:40:59,241 [238] [DEBUG] [app] Starting request: urn:request:03e6cb67-7c25-4aef-854e-967b6094fe1d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:40:59,241 [238] [DEBUG] [app] Ending request: urn:request:03e6cb67-7c25-4aef-854e-967b6094fe1d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03e6cb67-7c25-4aef-854e-967b6094fe1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:40:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:40:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:40:59,242 [236] [DEBUG] [app] Starting request: urn:request:c7cdca8d-36e8-4b72-9b85-d9708624a573 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:40:59,243 [236] [DEBUG] [app] Ending request: urn:request:c7cdca8d-36e8-4b72-9b85-d9708624a573 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c7cdca8d-36e8-4b72-9b85-d9708624a573', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:40:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:40:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:40:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:40:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:40:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:40:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:40:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:40:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:40:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:40:59,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:40:59,252 [237] [DEBUG] [app] Ending request: urn:request:a6c24168-2ad7-4c7e-b8e0-2b29068a25e7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6c24168-2ad7-4c7e-b8e0-2b29068a25e7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:59,253 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:40:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:40:59,253 [235] [DEBUG] [app] Ending request: urn:request:469c4c89-0af7-4ae3-b777-06f3cf1f457d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:469c4c89-0af7-4ae3-b777-06f3cf1f457d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:40:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:40:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:40:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:40:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:40:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:40:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:04.936688+00:00 (in 4.998259 seconds)
exportactionlogsworker stdout | 2025-02-07 05:40:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:59 UTC)" (scheduled at 2025-02-07 05:40:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:40:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:40:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:41:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:41:02,634 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:41:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:41:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:23.420886+00:00 (in 20.001308 seconds)
repositorygcworker stdout | 2025-02-07 05:41:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:03 UTC)" (scheduled at 2025-02-07 05:41:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:41:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:41:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 3, 419870), True, datetime.datetime(2025, 2, 7, 5, 41, 3, 419870), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:41:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:41:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:41:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:41:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:41:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:33.919286+00:00 (in 29.999535 seconds)
buildlogsarchiver stdout | 2025-02-07 05:41:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:33 UTC)" (scheduled at 2025-02-07 05:41:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:41:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 41, 3, 920033), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:41:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:41:03,928 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:41:03,928 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:41:04,303 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:41:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:41:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:59.937899+00:00 (in 55.000766 seconds)
exportactionlogsworker stdout | 2025-02-07 05:41:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:04 UTC)" (scheduled at 2025-02-07 05:41:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:41:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:41:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 4, 937469), True, datetime.datetime(2025, 2, 7, 5, 41, 4, 937469), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:41:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:41:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:41:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:41:05,468 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:41:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:41:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:18.732147+00:00 (in 9.999577 seconds)
notificationworker stdout | 2025-02-07 05:41:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:18 UTC)" (scheduled at 2025-02-07 05:41:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:41:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:41:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 8, 732835), True, datetime.datetime(2025, 2, 7, 5, 41, 8, 732835), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:41:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:41:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:41:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:41:09,722 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:41:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:41:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:24.423055+00:00 (in 13.998373 seconds)
gcworker stdout | 2025-02-07 05:41:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:40 UTC)" (scheduled at 2025-02-07 05:41:10.424250+00:00)
gcworker stdout | 2025-02-07 05:41:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:41:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906570430, None, 1, 0])
gcworker stdout | 2025-02-07 05:41:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:41:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:41:10,773 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:41:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:41:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:41.831521+00:00 (in 29.999565 seconds)
autopruneworker stdout | 2025-02-07 05:41:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:41 UTC)" (scheduled at 2025-02-07 05:41:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:41:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903271836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:41:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:41:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:41:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:41:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:41:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:29.225378+00:00 (in 14.999523 seconds)
quotatotalworker stdout | 2025-02-07 05:41:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:29 UTC)" (scheduled at 2025-02-07 05:41:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:41:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:41:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:41:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:41:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:41:14,235 [235] [DEBUG] [app] Starting request: urn:request:c9a1da2f-ff89-4856-afe0-d2ee29df9343 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:14,235 [238] [DEBUG] [app] Starting request: urn:request:4859b3ab-de22-416b-91a1-e042d470f1f1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:41:14,239 [239] [DEBUG] [app] Starting request: urn:request:fcc66986-d71a-47e7-89cd-fb483f29f34d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:14,239 [246] [DEBUG] [app] Starting request: urn:request:c4094652-0ba5-43e0-b6bb-d62f50b61cfe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:14,239 [239] [DEBUG] [app] Ending request: urn:request:fcc66986-d71a-47e7-89cd-fb483f29f34d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fcc66986-d71a-47e7-89cd-fb483f29f34d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:14,239 [246] [DEBUG] [app] Ending request: urn:request:c4094652-0ba5-43e0-b6bb-d62f50b61cfe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c4094652-0ba5-43e0-b6bb-d62f50b61cfe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:41:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:41:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:14,242 [238] [DEBUG] [app] Starting request: urn:request:643c3d7d-872c-4b84-9526-1a711202a9c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:14,243 [238] [DEBUG] [app] Ending request: urn:request:643c3d7d-872c-4b84-9526-1a711202a9c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:643c3d7d-872c-4b84-9526-1a711202a9c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:41:14,243 [236] [DEBUG] [app] Starting request: urn:request:a50ff007-e886-48bb-8512-2e7ca31f4f98 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:14,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:41:14,243 [236] [DEBUG] [app] Ending request: urn:request:a50ff007-e886-48bb-8512-2e7ca31f4f98 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a50ff007-e886-48bb-8512-2e7ca31f4f98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:41:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:41:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:14,255 [238] [DEBUG] [app] Ending request: urn:request:4859b3ab-de22-416b-91a1-e042d470f1f1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4859b3ab-de22-416b-91a1-e042d470f1f1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:14,255 [235] [DEBUG] [app] Ending request: urn:request:c9a1da2f-ff89-4856-afe0-d2ee29df9343 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9a1da2f-ff89-4856-afe0-d2ee29df9343', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:41:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:41:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:41:14,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:41:14,784 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:41:15,337 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:41:15,959 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:41:17,347 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:41:17,952 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:41:17,957 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:41:17,960 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:41:17,963 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:41:17,965 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:41:18,078 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:41:18,156 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:41:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:41:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:28.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 05:41:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:28 UTC)" (scheduled at 2025-02-07 05:41:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:41:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:41:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 18, 732899), True, datetime.datetime(2025, 2, 7, 5, 41, 18, 732899), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:41:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:41:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:41:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:41:18,968 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:41:18,971 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:41:18,974 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:41:19,482 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:19,674 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:41:20,092 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:41:20,181 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:41:20,422 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:41:20,431 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:41:20,434 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:41:20,438 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:41:20,441 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:41:20,447 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:41:20,455 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:41:20,458 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:41:20,460 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:41:20,484 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:41:20,818 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:41:21,197 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:41:21,515 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:41:21,560 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
namespacegcworker stdout | 2025-02-07 05:41:23,214 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:41:23,236 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
repositorygcworker stdout | 2025-02-07 05:41:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:41:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:03.419110+00:00 (in 39.997765 seconds)
repositorygcworker stdout | 2025-02-07 05:41:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:23 UTC)" (scheduled at 2025-02-07 05:41:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:41:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:41:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:23 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:41:23,640 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:41:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:41:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:40.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 05:41:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:54 UTC)" (scheduled at 2025-02-07 05:41:24.423055+00:00)
gcworker stdout | 2025-02-07 05:41:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:41:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:41:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:41:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:41:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:38.732147+00:00 (in 9.999591 seconds)
notificationworker stdout | 2025-02-07 05:41:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:38 UTC)" (scheduled at 2025-02-07 05:41:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:41:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:41:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 28, 732820), True, datetime.datetime(2025, 2, 7, 5, 41, 28, 732820), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:41:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:41:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:41:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:41:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:41:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:44.225378+00:00 (in 14.999527 seconds)
quotatotalworker stdout | 2025-02-07 05:41:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:44 UTC)" (scheduled at 2025-02-07 05:41:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:41:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:41:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:41:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:41:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:41:29,235 [236] [DEBUG] [app] Starting request: urn:request:5d51f7de-1267-4099-b057-5fd8dd6cc35d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:29,235 [235] [DEBUG] [app] Starting request: urn:request:b4a6c600-e538-4bf2-9fe0-49df5f13d875 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:29,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:41:29,237 [246] [DEBUG] [app] Starting request: urn:request:50749fb5-524d-4b42-8668-fdd3d862bf32 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:29,237 [245] [DEBUG] [app] Starting request: urn:request:229e7b6b-a911-43b1-b7f2-46f2e4573dc2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:29,238 [246] [DEBUG] [app] Ending request: urn:request:50749fb5-524d-4b42-8668-fdd3d862bf32 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:50749fb5-524d-4b42-8668-fdd3d862bf32', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:29,238 [245] [DEBUG] [app] Ending request: urn:request:229e7b6b-a911-43b1-b7f2-46f2e4573dc2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:229e7b6b-a911-43b1-b7f2-46f2e4573dc2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:41:29,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:41:29,238 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:29,238 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:29,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:29,241 [238] [DEBUG] [app] Starting request: urn:request:263cd44f-1b5b-4212-98a0-30a9675f91e8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:29,241 [235] [DEBUG] [app] Starting request: urn:request:b4cbf9f4-3ebf-4bf7-bbac-f45e1c28e8ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:29,241 [238] [DEBUG] [app] Ending request: urn:request:263cd44f-1b5b-4212-98a0-30a9675f91e8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:263cd44f-1b5b-4212-98a0-30a9675f91e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:41:29,241 [235] [DEBUG] [app] Ending request: urn:request:b4cbf9f4-3ebf-4bf7-bbac-f45e1c28e8ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4cbf9f4-3ebf-4bf7-bbac-f45e1c28e8ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:41:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:41:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:41:29,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:29,242 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:29,242 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:29,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:29,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:29,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:29,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:29,253 [236] [DEBUG] [app] Ending request: urn:request:5d51f7de-1267-4099-b057-5fd8dd6cc35d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d51f7de-1267-4099-b057-5fd8dd6cc35d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:29,253 [235] [DEBUG] [app] Ending request: urn:request:b4a6c600-e538-4bf2-9fe0-49df5f13d875 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b4a6c600-e538-4bf2-9fe0-49df5f13d875', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:29,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:41:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:41:29,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:41:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 05:41:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:41:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:32.625945+00:00 (in 1.004321 seconds)
namespacegcworker stdout | 2025-02-07 05:41:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:31 UTC)" (scheduled at 2025-02-07 05:41:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:41:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:41:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 31, 621897), True, datetime.datetime(2025, 2, 7, 5, 41, 31, 621897), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:41:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:41:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:41:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:41:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:41:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:13.624301+00:00 (in 40.997897 seconds)
namespacegcworker stdout | 2025-02-07 05:41:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:32 UTC)" (scheduled at 2025-02-07 05:41:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:41:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:41:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:41:32,670 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:41:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:41:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:03.919286+00:00 (in 29.999529 seconds)
buildlogsarchiver stdout | 2025-02-07 05:41:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:03 UTC)" (scheduled at 2025-02-07 05:41:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:41:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 41, 33, 920011), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:41:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:41:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:41:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:41:34,329 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:41:35,481 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:56.713547+00:00 (in 18.997155 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:37 UTC)" (scheduled at 2025-02-07 05:41:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:41:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:41:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:48.732147+00:00 (in 9.999586 seconds)
notificationworker stdout | 2025-02-07 05:41:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:48 UTC)" (scheduled at 2025-02-07 05:41:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:41:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:41:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 38, 732825), True, datetime.datetime(2025, 2, 7, 5, 41, 38, 732825), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:41:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:41:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:41:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:41:39,748 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:41:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:41:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:54.423055+00:00 (in 13.998396 seconds)
gcworker stdout | 2025-02-07 05:41:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:10 UTC)" (scheduled at 2025-02-07 05:41:40.424250+00:00)
gcworker stdout | 2025-02-07 05:41:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:41:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906600430, None, 1, 0])
gcworker stdout | 2025-02-07 05:41:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:41:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:41:40,809 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:41:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:41:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:11.831521+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2025-02-07 05:41:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:11 UTC)" (scheduled at 2025-02-07 05:41:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:41:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903301836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:41:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:41:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:41:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:41:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:41:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:59.225378+00:00 (in 14.999521 seconds)
quotatotalworker stdout | 2025-02-07 05:41:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:59 UTC)" (scheduled at 2025-02-07 05:41:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:41:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:41:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:41:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:41:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:41:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:41:44,236 [238] [DEBUG] [app] Starting request: urn:request:02d54281-79b4-4854-975b-e0325f32004d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:44,236 [235] [DEBUG] [app] Starting request: urn:request:d3e5c8dc-7eeb-48cf-a511-50b36bbd577f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:41:44,239 [246] [DEBUG] [app] Starting request: urn:request:eda0884f-fa95-488b-bf96-993d9eab2654 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:44,239 [239] [DEBUG] [app] Starting request: urn:request:3f92d05b-c8ea-4d1f-8712-1434407b0869 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:44,239 [246] [DEBUG] [app] Ending request: urn:request:eda0884f-fa95-488b-bf96-993d9eab2654 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eda0884f-fa95-488b-bf96-993d9eab2654', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:44,239 [239] [DEBUG] [app] Ending request: urn:request:3f92d05b-c8ea-4d1f-8712-1434407b0869 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f92d05b-c8ea-4d1f-8712-1434407b0869', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:41:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:41:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:41:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:44,242 [238] [DEBUG] [app] Starting request: urn:request:0eeef0a7-bfa9-4bed-a473-2ee05b6c2631 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:44,242 [238] [DEBUG] [app] Ending request: urn:request:0eeef0a7-bfa9-4bed-a473-2ee05b6c2631 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0eeef0a7-bfa9-4bed-a473-2ee05b6c2631', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:41:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:41:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:44,244 [236] [DEBUG] [app] Starting request: urn:request:56aaceba-16a7-4a39-8dd2-86948c2b95cb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:44,244 [236] [DEBUG] [app] Ending request: urn:request:56aaceba-16a7-4a39-8dd2-86948c2b95cb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:56aaceba-16a7-4a39-8dd2-86948c2b95cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:41:44,245 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:41:44,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:44,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:44,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:44,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:44,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:44,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:44,254 [235] [DEBUG] [app] Ending request: urn:request:d3e5c8dc-7eeb-48cf-a511-50b36bbd577f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d3e5c8dc-7eeb-48cf-a511-50b36bbd577f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:44,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:41:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:41:44,256 [238] [DEBUG] [app] Ending request: urn:request:02d54281-79b4-4854-975b-e0325f32004d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:02d54281-79b4-4854-975b-e0325f32004d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:44,256 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:41:44,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:41:44,803 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:41:45,360 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:41:45,966 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:41:47,353 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:41:47,960 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:41:47,963 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:41:47,967 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:41:47,969 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:41:47,972 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:41:48,114 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:41:48,193 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:41:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:41:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:50.814677+00:00 (in 2.082081 seconds)
notificationworker stdout | 2025-02-07 05:41:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:58 UTC)" (scheduled at 2025-02-07 05:41:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:41:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:41:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 48, 732915), True, datetime.datetime(2025, 2, 7, 5, 41, 48, 732915), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:41:48,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:41:48,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:41:48,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:41:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:41:48,976 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:41:48,981 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:41:48,984 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:41:49,495 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:49,685 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:41:50,129 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:41:50,217 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:41:50,430 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:41:50,444 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:41:50,447 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:41:50,450 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:41:50,452 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:41:50,456 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:41:50,468 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:41:50,471 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:41:50,474 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:41:50,518 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:41:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:41:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:41:58.732147+00:00 (in 7.917031 seconds)
notificationworker stdout | 2025-02-07 05:41:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:50 UTC)" (scheduled at 2025-02-07 05:41:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:41:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:41:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:50 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:41:50,853 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:41:51,206 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:41:51,523 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:41:51,597 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
manifestbackfillworker stdout | 2025-02-07 05:41:52,134 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2025-02-07 05:41:52,135 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 06:41:52.134599+00:00 (in 3599.999548 seconds)
manifestbackfillworker stdout | 2025-02-07 05:41:52,135 [68] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-02-07 06:41:52 UTC)" (scheduled at 2025-02-07 05:41:52.134599+00:00)
manifestbackfillworker stdout | 2025-02-07 05:41:52,135 [68] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0])
manifestbackfillworker stdout | 2025-02-07 05:41:52,141 [68] [DEBUG] [__main__] Manifest backfill worker has completed; skipping
manifestbackfillworker stdout | 2025-02-07 05:41:52,141 [68] [DEBUG] [data.database] Disconnecting from database.
manifestbackfillworker stdout | 2025-02-07 05:41:52,141 [68] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2025-02-07 06:41:52 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:41:53,245 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
buildlogsarchiver stdout | 2025-02-07 05:41:53,258 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
blobuploadcleanupworker stdout | 2025-02-07 05:41:53,671 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:41:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:41:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:10.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 05:41:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:24 UTC)" (scheduled at 2025-02-07 05:41:54.423055+00:00)
gcworker stdout | 2025-02-07 05:41:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:41:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:41:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:41:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:56.118855+00:00 (in 59.999501 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:41:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:56 UTC)" (scheduled at 2025-02-07 05:41:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:41:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:41:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:41:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:37.715878+00:00 (in 41.001898 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:56 UTC)" (scheduled at 2025-02-07 05:41:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:41:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:41:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:41:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:08.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 05:41:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:08 UTC)" (scheduled at 2025-02-07 05:41:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:41:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:41:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 41, 58, 732892), True, datetime.datetime(2025, 2, 7, 5, 41, 58, 732892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:41:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:41:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:41:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:41:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:41:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:14.225378+00:00 (in 14.999511 seconds)
quotatotalworker stdout | 2025-02-07 05:41:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:14 UTC)" (scheduled at 2025-02-07 05:41:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:41:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:41:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:41:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:41:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:41:59,235 [236] [DEBUG] [app] Starting request: urn:request:d869afaf-eba0-447b-934b-4ef6dd85c94f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:59,235 [238] [DEBUG] [app] Starting request: urn:request:ac1831eb-86cc-48a3-9de1-0b6cfa486f4e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:41:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:59,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:41:59,238 [239] [DEBUG] [app] Starting request: urn:request:9045fb8a-44f6-4a82-b307-14f9692da2cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:59,238 [244] [DEBUG] [app] Starting request: urn:request:489562bd-caa8-4e95-b68b-271b983395bf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:41:59,238 [239] [DEBUG] [app] Ending request: urn:request:9045fb8a-44f6-4a82-b307-14f9692da2cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9045fb8a-44f6-4a82-b307-14f9692da2cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:59,238 [244] [DEBUG] [app] Ending request: urn:request:489562bd-caa8-4e95-b68b-271b983395bf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:489562bd-caa8-4e95-b68b-271b983395bf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:41:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:41:59,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:41:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:41:59,242 [236] [DEBUG] [app] Starting request: urn:request:9fa7e75c-547c-48f1-8993-77d60976fd68 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:59,242 [236] [DEBUG] [app] Ending request: urn:request:9fa7e75c-547c-48f1-8993-77d60976fd68 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9fa7e75c-547c-48f1-8993-77d60976fd68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:41:59,242 [235] [DEBUG] [app] Starting request: urn:request:dbd287af-2bf3-4ce0-a2ff-5dedd9c5aa17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:41:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:41:59,242 [235] [DEBUG] [app] Ending request: urn:request:dbd287af-2bf3-4ce0-a2ff-5dedd9c5aa17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dbd287af-2bf3-4ce0-a2ff-5dedd9c5aa17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:41:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:41:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:41:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:41:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:41:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:41:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:41:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:41:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:41:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:41:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:41:59,254 [236] [DEBUG] [app] Ending request: urn:request:d869afaf-eba0-447b-934b-4ef6dd85c94f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d869afaf-eba0-447b-934b-4ef6dd85c94f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:59,254 [238] [DEBUG] [app] Ending request: urn:request:ac1831eb-86cc-48a3-9de1-0b6cfa486f4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac1831eb-86cc-48a3-9de1-0b6cfa486f4e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:41:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:41:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:41:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 05:41:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:41:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:41:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:41:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:41:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:04.936688+00:00 (in 4.998324 seconds)
exportactionlogsworker stdout | 2025-02-07 05:41:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:59 UTC)" (scheduled at 2025-02-07 05:41:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:41:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:41:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:42:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:42:02,688 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:42:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:42:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:23.420886+00:00 (in 20.001324 seconds)
repositorygcworker stdout | 2025-02-07 05:42:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:03 UTC)" (scheduled at 2025-02-07 05:42:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:42:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:42:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 3, 419840), True, datetime.datetime(2025, 2, 7, 5, 42, 3, 419840), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:42:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:42:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:42:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:42:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:42:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:33.919286+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2025-02-07 05:42:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:33 UTC)" (scheduled at 2025-02-07 05:42:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:42:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 42, 3, 920002), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:42:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:42:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:42:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:42:04,353 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:42:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:42:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:59.937899+00:00 (in 55.000781 seconds)
exportactionlogsworker stdout | 2025-02-07 05:42:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:04 UTC)" (scheduled at 2025-02-07 05:42:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:42:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:42:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 4, 937423), True, datetime.datetime(2025, 2, 7, 5, 42, 4, 937423), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:42:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:42:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:42:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:42:05,518 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:42:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:18.732147+00:00 (in 9.999581 seconds)
notificationworker stdout | 2025-02-07 05:42:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:18 UTC)" (scheduled at 2025-02-07 05:42:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:42:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:42:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 8, 732839), True, datetime.datetime(2025, 2, 7, 5, 42, 8, 732839), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:42:08,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:42:08,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:42:08,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:42:09,784 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:42:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:42:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:24.423055+00:00 (in 13.998360 seconds)
gcworker stdout | 2025-02-07 05:42:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:40 UTC)" (scheduled at 2025-02-07 05:42:10.424250+00:00)
gcworker stdout | 2025-02-07 05:42:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:42:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906630431, None, 1, 0])
gcworker stdout | 2025-02-07 05:42:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:42:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:42:10,846 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:42:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:42:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:41.831521+00:00 (in 29.999572 seconds)
autopruneworker stdout | 2025-02-07 05:42:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:41 UTC)" (scheduled at 2025-02-07 05:42:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:42:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903331836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:42:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:42:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:42:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:41 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:42:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:42:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:31.621131+00:00 (in 17.996382 seconds)
namespacegcworker stdout | 2025-02-07 05:42:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:47:13 UTC)" (scheduled at 2025-02-07 05:42:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:42:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 42, 13, 625002), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:42:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 42, 13, 625002), True, datetime.datetime(2025, 2, 7, 5, 42, 13, 625002), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:42:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 42, 13, 625002), True, datetime.datetime(2025, 2, 7, 5, 42, 13, 625002), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 42, 13, 625002), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:42:13,634 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:42:13,634 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:47:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:42:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:42:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:29.225378+00:00 (in 14.999548 seconds)
quotatotalworker stdout | 2025-02-07 05:42:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:29 UTC)" (scheduled at 2025-02-07 05:42:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:42:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:42:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:42:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:42:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:42:14,235 [238] [DEBUG] [app] Starting request: urn:request:bfcc2028-a5a3-4bd7-b58d-114ffc4bb82a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:14,235 [235] [DEBUG] [app] Starting request: urn:request:a0819637-672d-4bc4-94b9-af4cfce01807 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:42:14,238 [239] [DEBUG] [app] Starting request: urn:request:9eccab61-210f-4b6a-a282-182ca9f7dda9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:42:14,238 [239] [DEBUG] [app] Ending request: urn:request:9eccab61-210f-4b6a-a282-182ca9f7dda9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9eccab61-210f-4b6a-a282-182ca9f7dda9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:42:14,239 [246] [DEBUG] [app] Starting request: urn:request:6b28f02e-7d4a-46de-8443-4b799f8e284d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:42:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:42:14,239 [246] [DEBUG] [app] Ending request: urn:request:6b28f02e-7d4a-46de-8443-4b799f8e284d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6b28f02e-7d4a-46de-8443-4b799f8e284d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:42:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:42:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:42:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:42:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:14,241 [236] [DEBUG] [app] Starting request: urn:request:af282da7-7974-4189-919c-300e196c03c3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:14,241 [236] [DEBUG] [app] Ending request: urn:request:af282da7-7974-4189-919c-300e196c03c3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af282da7-7974-4189-919c-300e196c03c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:42:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:42:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:14,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:14,243 [237] [DEBUG] [app] Starting request: urn:request:8d88b51e-bfa5-4462-918d-eda335293414 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:14,243 [237] [DEBUG] [app] Ending request: urn:request:8d88b51e-bfa5-4462-918d-eda335293414 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d88b51e-bfa5-4462-918d-eda335293414', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:42:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:42:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:14,252 [235] [DEBUG] [app] Ending request: urn:request:a0819637-672d-4bc4-94b9-af4cfce01807 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0819637-672d-4bc4-94b9-af4cfce01807', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:42:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:14,254 [238] [DEBUG] [app] Ending request: urn:request:bfcc2028-a5a3-4bd7-b58d-114ffc4bb82a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bfcc2028-a5a3-4bd7-b58d-114ffc4bb82a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:42:14,840 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:42:15,396 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:42:15,980 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:42:17,381 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:42:17,968 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:42:17,971 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:42:17,975 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:42:17,978 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:42:17,980 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:42:18,124 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:42:18,217 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:42:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:28.732147+00:00 (in 9.999570 seconds)
notificationworker stdout | 2025-02-07 05:42:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:28 UTC)" (scheduled at 2025-02-07 05:42:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:42:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:42:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 18, 732890), True, datetime.datetime(2025, 2, 7, 5, 42, 18, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:42:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:42:18,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:42:18,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:42:18,985 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:42:18,987 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:42:18,990 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:42:19,518 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:19,718 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:42:20,165 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:42:20,251 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:42:20,443 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:42:20,454 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:42:20,457 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:42:20,461 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:42:20,464 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:42:20,466 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:42:20,477 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:42:20,480 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:42:20,482 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:42:20,554 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:42:20,876 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:42:21,234 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:42:21,559 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:42:21,618 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:42:23,274 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:42:23,279 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
repositorygcworker stdout | 2025-02-07 05:42:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:42:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:03.419110+00:00 (in 39.997741 seconds)
repositorygcworker stdout | 2025-02-07 05:42:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:23 UTC)" (scheduled at 2025-02-07 05:42:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:42:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:42:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:23 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:42:23,705 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:42:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:42:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:40.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 05:42:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:54 UTC)" (scheduled at 2025-02-07 05:42:24.423055+00:00)
gcworker stdout | 2025-02-07 05:42:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:42:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:42:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:42:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:33.814166+00:00 (in 5.081578 seconds)
notificationworker stdout | 2025-02-07 05:42:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:38 UTC)" (scheduled at 2025-02-07 05:42:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:42:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:42:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 28, 732874), True, datetime.datetime(2025, 2, 7, 5, 42, 28, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:42:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:42:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:42:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:42:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:42:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:44.225378+00:00 (in 14.999564 seconds)
quotatotalworker stdout | 2025-02-07 05:42:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:44 UTC)" (scheduled at 2025-02-07 05:42:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:42:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:42:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:42:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:42:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:42:29,236 [235] [DEBUG] [app] Starting request: urn:request:ce7549c6-e35a-4f80-a7f6-c5e127fb1913 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:29,236 [238] [DEBUG] [app] Starting request: urn:request:9a1463fe-017d-4dd8-9e30-349aafc1c717 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:29,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:42:29,239 [239] [DEBUG] [app] Starting request: urn:request:9669822e-a99f-49ce-81a2-a8ebd96f1af2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:42:29,239 [239] [DEBUG] [app] Ending request: urn:request:9669822e-a99f-49ce-81a2-a8ebd96f1af2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9669822e-a99f-49ce-81a2-a8ebd96f1af2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:42:29,240 [244] [DEBUG] [app] Starting request: urn:request:a844b245-0d7d-4730-a0f7-95261d24596b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:42:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:42:29,240 [244] [DEBUG] [app] Ending request: urn:request:a844b245-0d7d-4730-a0f7-95261d24596b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a844b245-0d7d-4730-a0f7-95261d24596b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:42:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:42:29,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:42:29,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:29,243 [236] [DEBUG] [app] Starting request: urn:request:4810f714-8cb1-4009-837e-498a04f99c81 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:29,243 [236] [DEBUG] [app] Ending request: urn:request:4810f714-8cb1-4009-837e-498a04f99c81 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4810f714-8cb1-4009-837e-498a04f99c81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:42:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:42:29,244 [238] [DEBUG] [app] Starting request: urn:request:925d7139-7f21-43be-aab3-930f78e9372b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:29,244 [238] [DEBUG] [app] Ending request: urn:request:925d7139-7f21-43be-aab3-930f78e9372b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:925d7139-7f21-43be-aab3-930f78e9372b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:42:29,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:42:29,245 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:29,245 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:29,245 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:29,245 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:29,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:29,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:29,249 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:29,249 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:29,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:29,254 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:29,255 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:29,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:29,256 [235] [DEBUG] [app] Ending request: urn:request:ce7549c6-e35a-4f80-a7f6-c5e127fb1913 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ce7549c6-e35a-4f80-a7f6-c5e127fb1913', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:29,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:29,257 [238] [DEBUG] [app] Ending request: urn:request:9a1463fe-017d-4dd8-9e30-349aafc1c717 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9a1463fe-017d-4dd8-9e30-349aafc1c717', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:29,257 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:29,257 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:42:29,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
namespacegcworker stdout | 2025-02-07 05:42:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:42:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:32.625945+00:00 (in 1.004342 seconds)
namespacegcworker stdout | 2025-02-07 05:42:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:31 UTC)" (scheduled at 2025-02-07 05:42:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:42:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:42:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 31, 621912), True, datetime.datetime(2025, 2, 7, 5, 42, 31, 621912), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:42:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:42:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:42:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:42:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:42:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:31.621131+00:00 (in 58.994733 seconds)
namespacegcworker stdout | 2025-02-07 05:42:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:32 UTC)" (scheduled at 2025-02-07 05:42:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:42:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:42:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:42:32,715 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 05:42:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:38.732147+00:00 (in 4.917561 seconds)
notificationworker stdout | 2025-02-07 05:42:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:47:33 UTC)" (scheduled at 2025-02-07 05:42:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:42:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 42, 33, 814829), 'notification/%'])
notificationworker stdout | 2025-02-07 05:42:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 42, 33, 814829), True, datetime.datetime(2025, 2, 7, 5, 42, 33, 814829), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:42:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 42, 33, 814829), True, datetime.datetime(2025, 2, 7, 5, 42, 33, 814829), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 42, 33, 814829), 'notification/%'])
notificationworker stdout | 2025-02-07 05:42:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:42:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:47:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:42:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:42:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:03.919286+00:00 (in 29.999557 seconds)
buildlogsarchiver stdout | 2025-02-07 05:42:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:03 UTC)" (scheduled at 2025-02-07 05:42:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:42:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 42, 33, 919975), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:42:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:42:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:42:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:42:34,365 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:42:35,554 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:56.713547+00:00 (in 18.997206 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:37 UTC)" (scheduled at 2025-02-07 05:42:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:37,722 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:37,722 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:42:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:48.732147+00:00 (in 9.999589 seconds)
notificationworker stdout | 2025-02-07 05:42:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:48 UTC)" (scheduled at 2025-02-07 05:42:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:42:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:42:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 38, 732862), True, datetime.datetime(2025, 2, 7, 5, 42, 38, 732862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:42:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:42:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:42:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:42:39,821 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:42:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:42:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:54.423055+00:00 (in 13.998379 seconds)
gcworker stdout | 2025-02-07 05:42:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:10 UTC)" (scheduled at 2025-02-07 05:42:40.424250+00:00)
gcworker stdout | 2025-02-07 05:42:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:42:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906660431, None, 1, 0])
gcworker stdout | 2025-02-07 05:42:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:42:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:42:40,867 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:42:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:42:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:11.831521+00:00 (in 29.999564 seconds)
autopruneworker stdout | 2025-02-07 05:42:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:11 UTC)" (scheduled at 2025-02-07 05:42:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:42:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903361836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:42:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:42:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:42:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:42:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:42:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:59.225378+00:00 (in 14.999431 seconds)
quotatotalworker stdout | 2025-02-07 05:42:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:59 UTC)" (scheduled at 2025-02-07 05:42:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:42:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:42:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:42:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:42:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:42:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:42:44,236 [235] [DEBUG] [app] Starting request: urn:request:e02212c5-749f-44f7-99dc-7992de34ee8f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:44,236 [236] [DEBUG] [app] Starting request: urn:request:6a6e61ba-5e48-47a1-b4ab-7969a6318f3e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:44,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:42:44,239 [246] [DEBUG] [app] Starting request: urn:request:6714cce4-8d92-467f-9d75-08a1678a07e7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:42:44,240 [245] [DEBUG] [app] Starting request: urn:request:0b33007b-6fcc-493e-8e19-57d0948073dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:42:44,240 [246] [DEBUG] [app] Ending request: urn:request:6714cce4-8d92-467f-9d75-08a1678a07e7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6714cce4-8d92-467f-9d75-08a1678a07e7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:42:44,240 [245] [DEBUG] [app] Ending request: urn:request:0b33007b-6fcc-493e-8e19-57d0948073dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b33007b-6fcc-493e-8e19-57d0948073dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:42:44,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:42:44,241 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:42:44,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:44,241 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:44,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:44,243 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:44,244 [238] [DEBUG] [app] Starting request: urn:request:02b53c83-f4b7-45cd-8243-989963364d80 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:44,244 [238] [DEBUG] [app] Ending request: urn:request:02b53c83-f4b7-45cd-8243-989963364d80 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02b53c83-f4b7-45cd-8243-989963364d80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:42:44,245 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:42:44,245 [237] [DEBUG] [app] Starting request: urn:request:1465fc49-f2dd-4b40-b525-a0d814d367eb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:44,245 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:44,245 [237] [DEBUG] [app] Ending request: urn:request:1465fc49-f2dd-4b40-b525-a0d814d367eb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1465fc49-f2dd-4b40-b525-a0d814d367eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:42:44,246 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:44,246 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:44,246 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:42:44,246 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:44,246 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:44,247 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:44,250 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:44,250 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:44,250 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:44,250 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:44,254 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:44,254 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:44,256 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:44,256 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:44,257 [235] [DEBUG] [app] Ending request: urn:request:e02212c5-749f-44f7-99dc-7992de34ee8f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e02212c5-749f-44f7-99dc-7992de34ee8f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:44,257 [236] [DEBUG] [app] Ending request: urn:request:6a6e61ba-5e48-47a1-b4ab-7969a6318f3e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a6e61ba-5e48-47a1-b4ab-7969a6318f3e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:44,257 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:44,257 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:44,257 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.023)
gunicorn-web stdout | 2025-02-07 05:42:44,258 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:42:44,863 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:42:45,433 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:42:45,988 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:42:47,418 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:42:47,977 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:42:47,980 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:42:47,983 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:42:47,985 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:42:47,988 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:42:48,141 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:42:48,225 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:42:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:50.814677+00:00 (in 2.082061 seconds)
notificationworker stdout | 2025-02-07 05:42:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:58 UTC)" (scheduled at 2025-02-07 05:42:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:42:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:42:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 48, 732921), True, datetime.datetime(2025, 2, 7, 5, 42, 48, 732921), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:42:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:42:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:42:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:42:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:42:48,992 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:42:48,995 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:42:48,998 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:42:49,524 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:49,754 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:42:50,181 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:42:50,287 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:42:50,454 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:42:50,464 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:42:50,468 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:42:50,471 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:42:50,474 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:42:50,477 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:42:50,487 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:42:50,490 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:42:50,492 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:42:50,590 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:42:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:42:58.732147+00:00 (in 7.916980 seconds)
notificationworker stdout | 2025-02-07 05:42:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:50 UTC)" (scheduled at 2025-02-07 05:42:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:42:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:42:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:50 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:42:50,897 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:42:51,271 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:42:51,578 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:42:51,640 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:42:53,310 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:42:53,314 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:42:53,742 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:42:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:42:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:10.424250+00:00 (in 16.000740 seconds)
gcworker stdout | 2025-02-07 05:42:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:24 UTC)" (scheduled at 2025-02-07 05:42:54.423055+00:00)
gcworker stdout | 2025-02-07 05:42:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:42:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:42:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:42:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:56.118855+00:00 (in 59.999555 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:42:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:56 UTC)" (scheduled at 2025-02-07 05:42:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:42:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:42:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:42:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:37.715878+00:00 (in 41.001899 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:56 UTC)" (scheduled at 2025-02-07 05:42:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:42:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:42:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:42:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:08.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 05:42:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:08 UTC)" (scheduled at 2025-02-07 05:42:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:42:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:42:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 42, 58, 732877), True, datetime.datetime(2025, 2, 7, 5, 42, 58, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:42:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:42:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:42:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:42:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:42:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:14.225378+00:00 (in 14.999512 seconds)
quotatotalworker stdout | 2025-02-07 05:42:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:14 UTC)" (scheduled at 2025-02-07 05:42:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:42:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:42:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:42:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:42:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:42:59,235 [235] [DEBUG] [app] Starting request: urn:request:5476460b-7f70-4713-8e49-93d1e5f79b6a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:59,235 [237] [DEBUG] [app] Starting request: urn:request:1ac734ad-7f03-4fc8-b03b-4bcef2d9e579 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:42:59,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:42:59,238 [239] [DEBUG] [app] Starting request: urn:request:72a12387-c106-40fc-a895-3ca7059e67a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:42:59,238 [245] [DEBUG] [app] Starting request: urn:request:0605176c-ab7f-4142-88ba-0aadc6a5ad50 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:42:59,239 [239] [DEBUG] [app] Ending request: urn:request:72a12387-c106-40fc-a895-3ca7059e67a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:72a12387-c106-40fc-a895-3ca7059e67a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:42:59,239 [245] [DEBUG] [app] Ending request: urn:request:0605176c-ab7f-4142-88ba-0aadc6a5ad50 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0605176c-ab7f-4142-88ba-0aadc6a5ad50', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:42:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:42:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:42:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:42:59,242 [237] [DEBUG] [app] Starting request: urn:request:4684133d-7245-4fd4-a528-492dfdf843aa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:59,242 [237] [DEBUG] [app] Ending request: urn:request:4684133d-7245-4fd4-a528-492dfdf843aa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4684133d-7245-4fd4-a528-492dfdf843aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:42:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:42:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:59,243 [236] [DEBUG] [app] Starting request: urn:request:7d9d4dd4-14c4-42ec-adac-65db156aad51 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:42:59,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:59,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:59,243 [236] [DEBUG] [app] Ending request: urn:request:7d9d4dd4-14c4-42ec-adac-65db156aad51 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d9d4dd4-14c4-42ec-adac-65db156aad51', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:42:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:42:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:42:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:42:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:42:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:42:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:42:59,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:59,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:42:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:42:59,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:42:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:42:59,254 [237] [DEBUG] [app] Ending request: urn:request:1ac734ad-7f03-4fc8-b03b-4bcef2d9e579 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ac734ad-7f03-4fc8-b03b-4bcef2d9e579', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:59,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:59,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:42:59,255 [235] [DEBUG] [app] Ending request: urn:request:5476460b-7f70-4713-8e49-93d1e5f79b6a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5476460b-7f70-4713-8e49-93d1e5f79b6a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:42:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:42:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:42:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:42:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
exportactionlogsworker stdout | 2025-02-07 05:42:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:42:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:04.936688+00:00 (in 4.998345 seconds)
exportactionlogsworker stdout | 2025-02-07 05:42:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:59 UTC)" (scheduled at 2025-02-07 05:42:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:42:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:42:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:43:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:43:02,751 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:43:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:43:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:23.420886+00:00 (in 20.001310 seconds)
repositorygcworker stdout | 2025-02-07 05:43:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:03 UTC)" (scheduled at 2025-02-07 05:43:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:43:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:43:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 3, 419784), True, datetime.datetime(2025, 2, 7, 5, 43, 3, 419784), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:43:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:43:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:43:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:43:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:43:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:33.919286+00:00 (in 29.999525 seconds)
buildlogsarchiver stdout | 2025-02-07 05:43:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:33 UTC)" (scheduled at 2025-02-07 05:43:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:43:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 43, 3, 920033), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:43:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:43:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:43:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:43:04,402 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:43:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:43:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:59.937899+00:00 (in 55.000780 seconds)
exportactionlogsworker stdout | 2025-02-07 05:43:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:04 UTC)" (scheduled at 2025-02-07 05:43:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:43:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:43:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 4, 937449), True, datetime.datetime(2025, 2, 7, 5, 43, 4, 937449), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:43:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:43:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:43:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:43:05,591 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:43:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:43:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:18.732147+00:00 (in 9.999588 seconds)
notificationworker stdout | 2025-02-07 05:43:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:18 UTC)" (scheduled at 2025-02-07 05:43:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:43:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:43:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 8, 732822), True, datetime.datetime(2025, 2, 7, 5, 43, 8, 732822), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:43:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:43:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:43:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:43:09,836 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:43:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:43:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:24.423055+00:00 (in 13.998264 seconds)
gcworker stdout | 2025-02-07 05:43:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:40 UTC)" (scheduled at 2025-02-07 05:43:10.424250+00:00)
gcworker stdout | 2025-02-07 05:43:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:43:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906690431, None, 1, 0])
gcworker stdout | 2025-02-07 05:43:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:43:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:43:10,903 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:43:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:43:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:41.831521+00:00 (in 29.999559 seconds)
autopruneworker stdout | 2025-02-07 05:43:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:41 UTC)" (scheduled at 2025-02-07 05:43:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:43:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903391836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:43:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:43:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:43:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:43:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:43:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:29.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 05:43:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:29 UTC)" (scheduled at 2025-02-07 05:43:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:43:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:43:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:43:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:43:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:43:14,235 [237] [DEBUG] [app] Starting request: urn:request:326790f1-1485-42e1-b0ab-ad5cf01d8bd7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:14,235 [238] [DEBUG] [app] Starting request: urn:request:ab96a56b-a596-4c8d-93cf-840bb825e891 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:43:14,238 [246] [DEBUG] [app] Starting request: urn:request:82599642-94f2-42bd-8f11-fdbdadd97084 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:14,238 [239] [DEBUG] [app] Starting request: urn:request:2572e375-25aa-456a-b38d-a980c4d223b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:14,239 [246] [DEBUG] [app] Ending request: urn:request:82599642-94f2-42bd-8f11-fdbdadd97084 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82599642-94f2-42bd-8f11-fdbdadd97084', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:43:14,239 [239] [DEBUG] [app] Ending request: urn:request:2572e375-25aa-456a-b38d-a980c4d223b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2572e375-25aa-456a-b38d-a980c4d223b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:43:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:43:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:14,242 [235] [DEBUG] [app] Starting request: urn:request:46f3f4df-18d4-48c9-ab50-10f1be1fd478 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:14,242 [237] [DEBUG] [app] Starting request: urn:request:53d6b2ab-5880-4c88-a5f3-7d01369f16de (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:14,242 [235] [DEBUG] [app] Ending request: urn:request:46f3f4df-18d4-48c9-ab50-10f1be1fd478 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:46f3f4df-18d4-48c9-ab50-10f1be1fd478', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:43:14,242 [237] [DEBUG] [app] Ending request: urn:request:53d6b2ab-5880-4c88-a5f3-7d01369f16de (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53d6b2ab-5880-4c88-a5f3-7d01369f16de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:43:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:43:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:14,254 [237] [DEBUG] [app] Ending request: urn:request:326790f1-1485-42e1-b0ab-ad5cf01d8bd7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:326790f1-1485-42e1-b0ab-ad5cf01d8bd7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:14,255 [238] [DEBUG] [app] Ending request: urn:request:ab96a56b-a596-4c8d-93cf-840bb825e891 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ab96a56b-a596-4c8d-93cf-840bb825e891', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:43:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:43:14,900 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:43:15,466 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:43:16,025 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:43:17,437 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:43:17,984 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:43:17,988 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:43:17,992 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:43:17,994 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:43:17,997 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:43:18,177 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:43:18,248 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:43:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:43:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:28.732147+00:00 (in 9.999581 seconds)
notificationworker stdout | 2025-02-07 05:43:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:28 UTC)" (scheduled at 2025-02-07 05:43:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:43:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:43:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 18, 732829), True, datetime.datetime(2025, 2, 7, 5, 43, 18, 732829), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:43:18,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:43:18,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:43:18,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:43:19,000 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:43:19,003 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:43:19,006 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:43:19,560 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:19,790 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:43:20,196 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:43:20,323 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:43:20,465 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:43:20,473 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:43:20,477 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:43:20,482 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:43:20,484 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:43:20,487 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:43:20,492 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:43:20,497 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:43:20,499 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:43:20,603 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:43:20,932 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:43:21,307 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:43:21,594 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:43:21,658 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:43:23,331 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:43:23,342 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
repositorygcworker stdout | 2025-02-07 05:43:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:43:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:32.420707+00:00 (in 8.999347 seconds)
repositorygcworker stdout | 2025-02-07 05:43:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:23 UTC)" (scheduled at 2025-02-07 05:43:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:43:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:43:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:23 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:43:23,775 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:43:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:43:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:40.424250+00:00 (in 16.000695 seconds)
gcworker stdout | 2025-02-07 05:43:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:54 UTC)" (scheduled at 2025-02-07 05:43:24.423055+00:00)
gcworker stdout | 2025-02-07 05:43:24,424 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 05:43:24,430 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:43:24,430 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:43:24,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:43:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:43:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:43:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:38.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 05:43:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:38 UTC)" (scheduled at 2025-02-07 05:43:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:43:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:43:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 28, 732843), True, datetime.datetime(2025, 2, 7, 5, 43, 28, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:43:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:43:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:43:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:43:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:43:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:44.225378+00:00 (in 14.999525 seconds)
quotatotalworker stdout | 2025-02-07 05:43:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:44 UTC)" (scheduled at 2025-02-07 05:43:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:43:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:43:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:43:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:43:29,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:43:29,235 [235] [DEBUG] [app] Starting request: urn:request:11b6d6d3-5235-4f06-8845-50ebc69e993b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:29,235 [238] [DEBUG] [app] Starting request: urn:request:d31b5e63-dda1-47fc-af19-d7ee8bd7888d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:43:29,239 [239] [DEBUG] [app] Starting request: urn:request:785d0c67-4539-44e3-940f-b665d3a764bb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:29,239 [246] [DEBUG] [app] Starting request: urn:request:8cee3a2e-fec5-46df-a094-f84ac56da5f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:29,240 [246] [DEBUG] [app] Ending request: urn:request:8cee3a2e-fec5-46df-a094-f84ac56da5f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8cee3a2e-fec5-46df-a094-f84ac56da5f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:43:29,240 [239] [DEBUG] [app] Ending request: urn:request:785d0c67-4539-44e3-940f-b665d3a764bb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:785d0c67-4539-44e3-940f-b665d3a764bb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:43:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:43:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:29,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:29,243 [236] [DEBUG] [app] Starting request: urn:request:3cfa723b-0e50-4f10-97dd-45cee015ee94 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:29,243 [237] [DEBUG] [app] Starting request: urn:request:fa7ebc59-634a-4a76-b242-f359f94ac61a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:29,243 [236] [DEBUG] [app] Ending request: urn:request:3cfa723b-0e50-4f10-97dd-45cee015ee94 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3cfa723b-0e50-4f10-97dd-45cee015ee94', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:43:29,243 [237] [DEBUG] [app] Ending request: urn:request:fa7ebc59-634a-4a76-b242-f359f94ac61a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa7ebc59-634a-4a76-b242-f359f94ac61a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:43:29,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:29,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:29,255 [235] [DEBUG] [app] Ending request: urn:request:11b6d6d3-5235-4f06-8845-50ebc69e993b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11b6d6d3-5235-4f06-8845-50ebc69e993b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:29,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:43:29,255 [238] [DEBUG] [app] Ending request: urn:request:d31b5e63-dda1-47fc-af19-d7ee8bd7888d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d31b5e63-dda1-47fc-af19-d7ee8bd7888d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 05:43:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:43:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:32.625945+00:00 (in 1.004331 seconds)
namespacegcworker stdout | 2025-02-07 05:43:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:31 UTC)" (scheduled at 2025-02-07 05:43:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:43:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:43:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 31, 621888), True, datetime.datetime(2025, 2, 7, 5, 43, 31, 621888), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:43:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:43:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:43:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:43:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:43:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:03.419110+00:00 (in 30.997949 seconds)
repositorygcworker stdout | 2025-02-07 05:43:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:48:32 UTC)" (scheduled at 2025-02-07 05:43:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:43:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 43, 32, 421439), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:43:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 43, 32, 421439), True, datetime.datetime(2025, 2, 7, 5, 43, 32, 421439), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:43:32,430 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 43, 32, 421439), True, datetime.datetime(2025, 2, 7, 5, 43, 32, 421439), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 43, 32, 421439), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:43:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:43:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:48:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:43:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:43:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:31.621131+00:00 (in 58.994747 seconds)
namespacegcworker stdout | 2025-02-07 05:43:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:32 UTC)" (scheduled at 2025-02-07 05:43:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:43:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:43:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:43:32,774 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:43:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:43:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:03.919286+00:00 (in 29.999571 seconds)
buildlogsarchiver stdout | 2025-02-07 05:43:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:03 UTC)" (scheduled at 2025-02-07 05:43:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:43:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 43, 33, 919955), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:43:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:43:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:43:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:43:34,436 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:43:35,626 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:56.713547+00:00 (in 18.997211 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:37 UTC)" (scheduled at 2025-02-07 05:43:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:43:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:43:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:48.732147+00:00 (in 9.999597 seconds)
notificationworker stdout | 2025-02-07 05:43:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:48 UTC)" (scheduled at 2025-02-07 05:43:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:43:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:43:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 38, 732801), True, datetime.datetime(2025, 2, 7, 5, 43, 38, 732801), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:43:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:43:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:43:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:43:39,843 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:43:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:43:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:54.423055+00:00 (in 13.998330 seconds)
gcworker stdout | 2025-02-07 05:43:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:10 UTC)" (scheduled at 2025-02-07 05:43:40.424250+00:00)
gcworker stdout | 2025-02-07 05:43:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:43:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906720431, None, 1, 0])
gcworker stdout | 2025-02-07 05:43:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:43:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:43:40,932 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:43:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:43:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:11.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 05:43:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:11 UTC)" (scheduled at 2025-02-07 05:43:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:43:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903421836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:43:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:43:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:43:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:43:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:43:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:59.225378+00:00 (in 14.999501 seconds)
quotatotalworker stdout | 2025-02-07 05:43:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:59 UTC)" (scheduled at 2025-02-07 05:43:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:43:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:43:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:43:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:43:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:43:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:43:44,235 [237] [DEBUG] [app] Starting request: urn:request:7a33770f-9716-4577-9cef-e150f42368ff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:44,235 [236] [DEBUG] [app] Starting request: urn:request:9d4bce7d-e4e3-4d37-9b3e-cf7ba375980b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:43:44,238 [245] [DEBUG] [app] Starting request: urn:request:c3a8dfbd-7d18-4ede-b003-bdc99db3a053 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:44,238 [246] [DEBUG] [app] Starting request: urn:request:5f4acba0-0e6d-4565-927c-a30b62cfff97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:44,239 [246] [DEBUG] [app] Ending request: urn:request:5f4acba0-0e6d-4565-927c-a30b62cfff97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5f4acba0-0e6d-4565-927c-a30b62cfff97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:43:44,239 [245] [DEBUG] [app] Ending request: urn:request:c3a8dfbd-7d18-4ede-b003-bdc99db3a053 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3a8dfbd-7d18-4ede-b003-bdc99db3a053', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:43:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:43:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:44,242 [235] [DEBUG] [app] Starting request: urn:request:87c164cf-2a79-4ada-a7d0-79147b3cc0ff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:44,242 [236] [DEBUG] [app] Starting request: urn:request:3b14f1cd-f6dd-425f-93da-5bc0e5199edc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:44,242 [235] [DEBUG] [app] Ending request: urn:request:87c164cf-2a79-4ada-a7d0-79147b3cc0ff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:87c164cf-2a79-4ada-a7d0-79147b3cc0ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:43:44,242 [236] [DEBUG] [app] Ending request: urn:request:3b14f1cd-f6dd-425f-93da-5bc0e5199edc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b14f1cd-f6dd-425f-93da-5bc0e5199edc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:43:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:43:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:44,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:44,254 [237] [DEBUG] [app] Ending request: urn:request:7a33770f-9716-4577-9cef-e150f42368ff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a33770f-9716-4577-9cef-e150f42368ff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:44,254 [236] [DEBUG] [app] Ending request: urn:request:9d4bce7d-e4e3-4d37-9b3e-cf7ba375980b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d4bce7d-e4e3-4d37-9b3e-cf7ba375980b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:44,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:43:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:43:44,934 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:43:45,478 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:43:46,061 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:43:47,446 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:43:47,993 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:43:47,996 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:43:47,999 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:43:48,002 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:43:48,004 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:43:48,206 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:43:48,269 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:43:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:43:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:50.814677+00:00 (in 2.082170 seconds)
notificationworker stdout | 2025-02-07 05:43:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:58 UTC)" (scheduled at 2025-02-07 05:43:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:43:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:43:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 48, 732766), True, datetime.datetime(2025, 2, 7, 5, 43, 48, 732766), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:43:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:43:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:43:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:43:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:43:49,009 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:43:49,011 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:43:49,014 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:43:49,597 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:49,817 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:43:50,231 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:43:50,356 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:43:50,475 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:43:50,479 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:43:50,484 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:43:50,488 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:43:50,491 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:43:50,494 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:43:50,500 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:43:50,507 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:43:50,509 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:43:50,626 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:43:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:43:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:43:58.732147+00:00 (in 7.917037 seconds)
notificationworker stdout | 2025-02-07 05:43:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:50 UTC)" (scheduled at 2025-02-07 05:43:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:43:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:43:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:50 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:43:50,945 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:43:51,343 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:43:51,601 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:43:51,674 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:43:53,368 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:43:53,378 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:43:53,811 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:43:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:43:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:10.424250+00:00 (in 16.000723 seconds)
gcworker stdout | 2025-02-07 05:43:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:24 UTC)" (scheduled at 2025-02-07 05:43:54.423055+00:00)
gcworker stdout | 2025-02-07 05:43:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:43:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:43:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:43:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:56.118855+00:00 (in 59.999526 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:43:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:56 UTC)" (scheduled at 2025-02-07 05:43:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:43:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:43:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:43:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:37.715878+00:00 (in 41.001890 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:56 UTC)" (scheduled at 2025-02-07 05:43:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:43:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:43:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:43:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:08.732147+00:00 (in 9.999593 seconds)
notificationworker stdout | 2025-02-07 05:43:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:08 UTC)" (scheduled at 2025-02-07 05:43:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:43:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:43:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 43, 58, 732836), True, datetime.datetime(2025, 2, 7, 5, 43, 58, 732836), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:43:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:43:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:43:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:43:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:43:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:14.225378+00:00 (in 14.999567 seconds)
quotatotalworker stdout | 2025-02-07 05:43:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:14 UTC)" (scheduled at 2025-02-07 05:43:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:43:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:43:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:43:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:43:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:43:59,235 [235] [DEBUG] [app] Starting request: urn:request:3d093af1-de45-402a-b8cd-ac2c2969833c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:59,235 [238] [DEBUG] [app] Starting request: urn:request:94f48c9d-4c22-42e7-92f2-7a6eff7721ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:43:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:43:59,238 [239] [DEBUG] [app] Starting request: urn:request:1819e490-0000-424a-95b9-252a24162142 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:59,238 [246] [DEBUG] [app] Starting request: urn:request:b7d0086e-bb1e-412d-9d31-7059cfeb968e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:43:59,238 [239] [DEBUG] [app] Ending request: urn:request:1819e490-0000-424a-95b9-252a24162142 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1819e490-0000-424a-95b9-252a24162142', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:43:59,239 [246] [DEBUG] [app] Ending request: urn:request:b7d0086e-bb1e-412d-9d31-7059cfeb968e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b7d0086e-bb1e-412d-9d31-7059cfeb968e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:43:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:43:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:43:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:43:59,241 [237] [DEBUG] [app] Starting request: urn:request:d7c232b2-2367-45b0-aa88-09888321a766 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:59,241 [237] [DEBUG] [app] Ending request: urn:request:d7c232b2-2367-45b0-aa88-09888321a766 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d7c232b2-2367-45b0-aa88-09888321a766', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:43:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:59,242 [235] [DEBUG] [app] Starting request: urn:request:f12e5f7b-9a31-4c49-b280-f182efe20b80 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:43:59,242 [235] [DEBUG] [app] Ending request: urn:request:f12e5f7b-9a31-4c49-b280-f182efe20b80 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f12e5f7b-9a31-4c49-b280-f182efe20b80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:43:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:43:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:43:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:43:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:43:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:59,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:43:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:43:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:59,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:43:59,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:43:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:59,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:43:59,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:43:59,254 [238] [DEBUG] [app] Ending request: urn:request:94f48c9d-4c22-42e7-92f2-7a6eff7721ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94f48c9d-4c22-42e7-92f2-7a6eff7721ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:59,254 [235] [DEBUG] [app] Ending request: urn:request:3d093af1-de45-402a-b8cd-ac2c2969833c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3d093af1-de45-402a-b8cd-ac2c2969833c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:43:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:59,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:43:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:43:59,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:43:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:43:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:43:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:43:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:04.936688+00:00 (in 4.998333 seconds)
exportactionlogsworker stdout | 2025-02-07 05:43:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:59 UTC)" (scheduled at 2025-02-07 05:43:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:43:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:43:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:44:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:44:02,811 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:44:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:44:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:23.420886+00:00 (in 20.001264 seconds)
repositorygcworker stdout | 2025-02-07 05:44:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:03 UTC)" (scheduled at 2025-02-07 05:44:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:44:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:44:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 3, 419923), True, datetime.datetime(2025, 2, 7, 5, 44, 3, 419923), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:44:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:44:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:44:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:44:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:44:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:33.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 05:44:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:33 UTC)" (scheduled at 2025-02-07 05:44:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:44:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 44, 3, 920001), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:44:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:44:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:44:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:44:04,472 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:44:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:44:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:52.937648+00:00 (in 48.000512 seconds)
exportactionlogsworker stdout | 2025-02-07 05:44:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:04 UTC)" (scheduled at 2025-02-07 05:44:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:44:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:44:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 4, 937427), True, datetime.datetime(2025, 2, 7, 5, 44, 4, 937427), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:44:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:44:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:44:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:44:05,639 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:44:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:44:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:18.732147+00:00 (in 9.999548 seconds)
notificationworker stdout | 2025-02-07 05:44:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:18 UTC)" (scheduled at 2025-02-07 05:44:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:44:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:44:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 8, 732921), True, datetime.datetime(2025, 2, 7, 5, 44, 8, 732921), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:44:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:44:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:44:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:44:09,856 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:44:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:44:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:24.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 05:44:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:40 UTC)" (scheduled at 2025-02-07 05:44:10.424250+00:00)
gcworker stdout | 2025-02-07 05:44:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:44:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906750430, None, 1, 0])
gcworker stdout | 2025-02-07 05:44:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:44:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:44:10,969 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:44:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:44:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:41.831521+00:00 (in 29.999547 seconds)
autopruneworker stdout | 2025-02-07 05:44:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:41 UTC)" (scheduled at 2025-02-07 05:44:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:44:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903451836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:44:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:44:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:44:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:44:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:44:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:29.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 05:44:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:29 UTC)" (scheduled at 2025-02-07 05:44:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:44:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:44:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:44:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:44:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:44:14,235 [237] [DEBUG] [app] Starting request: urn:request:05a969d0-e81a-41c8-a848-61f4bd8cf4f7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:14,235 [238] [DEBUG] [app] Starting request: urn:request:c54f92ab-835e-4460-940a-beb4b936da80 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:44:14,238 [239] [DEBUG] [app] Starting request: urn:request:b757920d-364a-483d-89d2-8bac795fa6d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:44:14,239 [239] [DEBUG] [app] Ending request: urn:request:b757920d-364a-483d-89d2-8bac795fa6d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b757920d-364a-483d-89d2-8bac795fa6d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:44:14,239 [244] [DEBUG] [app] Starting request: urn:request:ad8e1d0a-4685-4e99-a2f7-c31ca51beecd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:44:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:44:14,239 [244] [DEBUG] [app] Ending request: urn:request:ad8e1d0a-4685-4e99-a2f7-c31ca51beecd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ad8e1d0a-4685-4e99-a2f7-c31ca51beecd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:44:14,240 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:44:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:14,242 [235] [DEBUG] [app] Starting request: urn:request:270a9196-d5ee-479e-9a57-e5789330ee48 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:44:14,242 [235] [DEBUG] [app] Ending request: urn:request:270a9196-d5ee-479e-9a57-e5789330ee48 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:270a9196-d5ee-479e-9a57-e5789330ee48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:44:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:44:14,243 [237] [DEBUG] [app] Starting request: urn:request:f8258e3f-ef64-4ca7-8521-d4fc0a94db85 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:44:14,243 [237] [DEBUG] [app] Ending request: urn:request:f8258e3f-ef64-4ca7-8521-d4fc0a94db85 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f8258e3f-ef64-4ca7-8521-d4fc0a94db85', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:44:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:14,254 [237] [DEBUG] [app] Ending request: urn:request:05a969d0-e81a-41c8-a848-61f4bd8cf4f7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05a969d0-e81a-41c8-a848-61f4bd8cf4f7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:14,254 [238] [DEBUG] [app] Ending request: urn:request:c54f92ab-835e-4460-940a-beb4b936da80 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c54f92ab-835e-4460-940a-beb4b936da80', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:44:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:44:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:44:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:44:14,965 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:44:15,514 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:44:16,088 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:44:17,482 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:44:18,001 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:44:18,003 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:44:18,006 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:44:18,009 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:44:18,012 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:44:18,236 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:44:18,283 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:44:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:44:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:28.732147+00:00 (in 9.999554 seconds)
notificationworker stdout | 2025-02-07 05:44:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:28 UTC)" (scheduled at 2025-02-07 05:44:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:44:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:44:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 18, 732911), True, datetime.datetime(2025, 2, 7, 5, 44, 18, 732911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:44:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:44:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:44:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:44:19,017 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:44:19,020 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:44:19,023 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:44:19,633 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:19,835 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:44:20,249 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:44:20,392 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:44:20,484 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:44:20,487 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:44:20,493 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:44:20,495 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:44:20,503 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:44:20,507 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:44:20,509 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:44:20,518 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:44:20,522 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:44:20,640 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:44:20,961 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:44:21,370 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:44:21,639 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:44:21,710 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:44:23,401 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:44:23,415 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
repositorygcworker stdout | 2025-02-07 05:44:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:44:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:03.419110+00:00 (in 39.997714 seconds)
repositorygcworker stdout | 2025-02-07 05:44:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:23 UTC)" (scheduled at 2025-02-07 05:44:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:44:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:44:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:23 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:44:23,843 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:44:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:44:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:40.424250+00:00 (in 16.000696 seconds)
gcworker stdout | 2025-02-07 05:44:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:54 UTC)" (scheduled at 2025-02-07 05:44:24.423055+00:00)
gcworker stdout | 2025-02-07 05:44:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:44:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:44:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:44:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:44:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:38.732147+00:00 (in 9.999591 seconds)
notificationworker stdout | 2025-02-07 05:44:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:38 UTC)" (scheduled at 2025-02-07 05:44:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:44:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:44:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 28, 732890), True, datetime.datetime(2025, 2, 7, 5, 44, 28, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:44:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:44:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:44:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:44:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:44:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:44.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 05:44:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:44 UTC)" (scheduled at 2025-02-07 05:44:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:44:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:44:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:44:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:44:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:44:29,235 [238] [DEBUG] [app] Starting request: urn:request:aae747e0-3906-4f2e-951f-881a2891543e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:29,235 [236] [DEBUG] [app] Starting request: urn:request:05ef09ba-d0f0-418a-8441-96402f789c9f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:44:29,239 [246] [DEBUG] [app] Starting request: urn:request:41b33ef1-6e12-4653-b6d5-2905dd94625c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:44:29,239 [239] [DEBUG] [app] Starting request: urn:request:16f77522-882d-4ba3-b2e1-83b45c3f9ec5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:44:29,239 [246] [DEBUG] [app] Ending request: urn:request:41b33ef1-6e12-4653-b6d5-2905dd94625c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:41b33ef1-6e12-4653-b6d5-2905dd94625c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:44:29,239 [239] [DEBUG] [app] Ending request: urn:request:16f77522-882d-4ba3-b2e1-83b45c3f9ec5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:16f77522-882d-4ba3-b2e1-83b45c3f9ec5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:44:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:44:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:44:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:29,242 [238] [DEBUG] [app] Starting request: urn:request:28bb8f04-b27e-4b4d-9c46-3e71aede802e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:44:29,242 [235] [DEBUG] [app] Starting request: urn:request:82d7213b-939d-4aa8-a882-a0ed37a7260a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:44:29,242 [238] [DEBUG] [app] Ending request: urn:request:28bb8f04-b27e-4b4d-9c46-3e71aede802e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:28bb8f04-b27e-4b4d-9c46-3e71aede802e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:29,243 [235] [DEBUG] [app] Ending request: urn:request:82d7213b-939d-4aa8-a882-a0ed37a7260a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82d7213b-939d-4aa8-a882-a0ed37a7260a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:44:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:44:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:29,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:29,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:29,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:29,254 [238] [DEBUG] [app] Ending request: urn:request:aae747e0-3906-4f2e-951f-881a2891543e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aae747e0-3906-4f2e-951f-881a2891543e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:29,254 [236] [DEBUG] [app] Ending request: urn:request:05ef09ba-d0f0-418a-8441-96402f789c9f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05ef09ba-d0f0-418a-8441-96402f789c9f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:44:29,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:44:29,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:44:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 05:44:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:44:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:32.625945+00:00 (in 1.004372 seconds)
namespacegcworker stdout | 2025-02-07 05:44:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:31 UTC)" (scheduled at 2025-02-07 05:44:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:44:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:44:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 31, 621852), True, datetime.datetime(2025, 2, 7, 5, 44, 31, 621852), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:44:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:44:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:44:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:44:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:44:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:31.621131+00:00 (in 58.994704 seconds)
namespacegcworker stdout | 2025-02-07 05:44:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:32 UTC)" (scheduled at 2025-02-07 05:44:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:44:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:44:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:44:32,843 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:44:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:44:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:03.919286+00:00 (in 29.999562 seconds)
buildlogsarchiver stdout | 2025-02-07 05:44:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:03 UTC)" (scheduled at 2025-02-07 05:44:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:44:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 44, 33, 919968), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:44:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:44:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:44:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:44:34,498 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:44:35,675 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:56.713547+00:00 (in 18.997157 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:37 UTC)" (scheduled at 2025-02-07 05:44:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:44:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:44:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:48.732147+00:00 (in 9.999542 seconds)
notificationworker stdout | 2025-02-07 05:44:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:48 UTC)" (scheduled at 2025-02-07 05:44:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:44:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:44:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 38, 732936), True, datetime.datetime(2025, 2, 7, 5, 44, 38, 732936), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:44:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:44:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:44:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:44:39,892 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:44:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:44:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:54.423055+00:00 (in 13.998313 seconds)
gcworker stdout | 2025-02-07 05:44:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:10 UTC)" (scheduled at 2025-02-07 05:44:40.424250+00:00)
gcworker stdout | 2025-02-07 05:44:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:44:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906780431, None, 1, 0])
gcworker stdout | 2025-02-07 05:44:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:44:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:44:40,992 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:44:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:44:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:11.831521+00:00 (in 29.999573 seconds)
autopruneworker stdout | 2025-02-07 05:44:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:11 UTC)" (scheduled at 2025-02-07 05:44:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:44:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903481836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:44:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:44:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:44:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:44:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:44:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:59.225378+00:00 (in 14.999540 seconds)
quotatotalworker stdout | 2025-02-07 05:44:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:59 UTC)" (scheduled at 2025-02-07 05:44:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:44:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:44:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:44:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:44:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:44:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:44:44,236 [237] [DEBUG] [app] Starting request: urn:request:94fdef23-b1ee-460e-a1a5-894a0dcfaf7d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:44,236 [235] [DEBUG] [app] Starting request: urn:request:26598e41-8bba-4cb3-860c-06423782ce9b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:44,238 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:44:44,239 [245] [DEBUG] [app] Starting request: urn:request:09acb90c-c43e-42ca-a56e-3b3e19a9bcd9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:44:44,239 [245] [DEBUG] [app] Ending request: urn:request:09acb90c-c43e-42ca-a56e-3b3e19a9bcd9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:09acb90c-c43e-42ca-a56e-3b3e19a9bcd9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:44:44,239 [239] [DEBUG] [app] Starting request: urn:request:6f87c87a-4d5a-4a2f-88a7-13f73acb96a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:44:44,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:44:44,240 [239] [DEBUG] [app] Ending request: urn:request:6f87c87a-4d5a-4a2f-88a7-13f73acb96a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f87c87a-4d5a-4a2f-88a7-13f73acb96a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:44:44,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:44:44,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:44,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:44,242 [235] [DEBUG] [app] Starting request: urn:request:abb18e8f-b0bf-414c-a159-9e9466d72a35 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:44:44,242 [235] [DEBUG] [app] Ending request: urn:request:abb18e8f-b0bf-414c-a159-9e9466d72a35 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:abb18e8f-b0bf-414c-a159-9e9466d72a35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:44:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:44:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:44,244 [238] [DEBUG] [app] Starting request: urn:request:a586405a-5951-452d-9bab-33061e8635f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:44:44,244 [238] [DEBUG] [app] Ending request: urn:request:a586405a-5951-452d-9bab-33061e8635f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a586405a-5951-452d-9bab-33061e8635f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:44,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:44:44,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:44,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:44,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:44,249 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:44,249 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:44,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:44,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:44,254 [235] [DEBUG] [app] Ending request: urn:request:26598e41-8bba-4cb3-860c-06423782ce9b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26598e41-8bba-4cb3-860c-06423782ce9b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:44:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:44:44,255 [237] [DEBUG] [app] Ending request: urn:request:94fdef23-b1ee-460e-a1a5-894a0dcfaf7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:94fdef23-b1ee-460e-a1a5-894a0dcfaf7d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:44,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:44:44,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:44:45,001 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:44:45,550 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:44:46,099 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:44:47,498 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:44:48,008 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:44:48,012 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:44:48,016 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:44:48,019 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:44:48,021 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:44:48,272 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:44:48,293 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:44:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:44:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:50.814677+00:00 (in 2.082101 seconds)
notificationworker stdout | 2025-02-07 05:44:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:58 UTC)" (scheduled at 2025-02-07 05:44:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:44:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:44:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 48, 732846), True, datetime.datetime(2025, 2, 7, 5, 44, 48, 732846), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:44:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:44:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:44:48,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:44:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:44:49,024 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:44:49,027 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:44:49,030 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:44:49,658 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:49,862 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:44:50,270 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:44:50,429 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:44:50,493 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:44:50,495 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:44:50,499 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:44:50,505 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:44:50,514 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:44:50,517 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:44:50,519 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:44:50,528 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:44:50,531 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:44:50,647 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:44:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:44:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:58.732147+00:00 (in 7.917014 seconds)
notificationworker stdout | 2025-02-07 05:44:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:50 UTC)" (scheduled at 2025-02-07 05:44:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:44:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:44:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:50 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:44:50,980 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:44:51,407 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:44:51,675 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:44:51,734 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
exportactionlogsworker stdout | 2025-02-07 05:44:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:44:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:44:59.937899+00:00 (in 6.999827 seconds)
exportactionlogsworker stdout | 2025-02-07 05:44:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:49:52 UTC)" (scheduled at 2025-02-07 05:44:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:44:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 44, 52, 938334), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:44:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 44, 52, 938334), True, datetime.datetime(2025, 2, 7, 5, 44, 52, 938334), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:44:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 44, 52, 938334), True, datetime.datetime(2025, 2, 7, 5, 44, 52, 938334), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 44, 52, 938334), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:44:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:44:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:49:52 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:44:53,437 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:44:53,450 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:44:53,863 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:44:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:44:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:10.424250+00:00 (in 16.000733 seconds)
gcworker stdout | 2025-02-07 05:44:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:24 UTC)" (scheduled at 2025-02-07 05:44:54.423055+00:00)
gcworker stdout | 2025-02-07 05:44:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:44:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:44:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:44:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:56.118855+00:00 (in 59.999539 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:44:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:56 UTC)" (scheduled at 2025-02-07 05:44:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:44:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:44:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:44:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:37.715878+00:00 (in 41.001889 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:56 UTC)" (scheduled at 2025-02-07 05:44:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:44:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:44:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:44:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:08.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 05:44:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:08 UTC)" (scheduled at 2025-02-07 05:44:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:44:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:44:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 44, 58, 732887), True, datetime.datetime(2025, 2, 7, 5, 44, 58, 732887), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:44:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:44:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:44:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:44:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:44:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:14.225378+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2025-02-07 05:44:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:14 UTC)" (scheduled at 2025-02-07 05:44:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:44:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:44:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:44:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:44:59,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:44:59,236 [237] [DEBUG] [app] Starting request: urn:request:ccc5b86d-c286-42ec-a2a7-558a8fb8a8bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:59,236 [235] [DEBUG] [app] Starting request: urn:request:507683a1-2251-4443-ad29-5f6824b4b290 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:44:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:44:59,239 [239] [DEBUG] [app] Starting request: urn:request:d341466a-8261-4667-9c06-3f68fbe54ca4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:44:59,239 [245] [DEBUG] [app] Starting request: urn:request:85376feb-68ff-4c7f-a50e-f52428a12306 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:44:59,239 [239] [DEBUG] [app] Ending request: urn:request:d341466a-8261-4667-9c06-3f68fbe54ca4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d341466a-8261-4667-9c06-3f68fbe54ca4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:44:59,239 [245] [DEBUG] [app] Ending request: urn:request:85376feb-68ff-4c7f-a50e-f52428a12306 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:85376feb-68ff-4c7f-a50e-f52428a12306', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:44:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:44:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:44:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:44:59,242 [238] [DEBUG] [app] Starting request: urn:request:b53eab9f-6266-4b3a-a14b-0697da113915 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:44:59,243 [238] [DEBUG] [app] Ending request: urn:request:b53eab9f-6266-4b3a-a14b-0697da113915 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b53eab9f-6266-4b3a-a14b-0697da113915', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:59,243 [235] [DEBUG] [app] Starting request: urn:request:30584b9f-c253-4078-bf01-9a7937bd6365 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:44:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:44:59,243 [235] [DEBUG] [app] Ending request: urn:request:30584b9f-c253-4078-bf01-9a7937bd6365 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:30584b9f-c253-4078-bf01-9a7937bd6365', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:44:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:44:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:44:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:44:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:44:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:44:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:44:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:44:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:44:59,253 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:44:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:44:59,255 [235] [DEBUG] [app] Ending request: urn:request:507683a1-2251-4443-ad29-5f6824b4b290 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:507683a1-2251-4443-ad29-5f6824b4b290', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:59,256 [237] [DEBUG] [app] Ending request: urn:request:ccc5b86d-c286-42ec-a2a7-558a8fb8a8bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ccc5b86d-c286-42ec-a2a7-558a8fb8a8bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:44:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:44:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:44:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:44:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:44:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:44:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:44:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:44:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:04.936688+00:00 (in 4.998303 seconds)
exportactionlogsworker stdout | 2025-02-07 05:44:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:59 UTC)" (scheduled at 2025-02-07 05:44:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:44:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:44:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:45:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:45:02,866 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:45:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:45:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:23.420886+00:00 (in 20.001328 seconds)
repositorygcworker stdout | 2025-02-07 05:45:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:03 UTC)" (scheduled at 2025-02-07 05:45:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:45:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:45:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 3, 419821), True, datetime.datetime(2025, 2, 7, 5, 45, 3, 419821), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:45:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:45:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:45:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:45:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:45:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:33.919286+00:00 (in 29.999483 seconds)
buildlogsarchiver stdout | 2025-02-07 05:45:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:33 UTC)" (scheduled at 2025-02-07 05:45:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:45:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 45, 3, 920093), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:45:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:45:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:45:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:45:04,520 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:45:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:45:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:59.937899+00:00 (in 55.000797 seconds)
exportactionlogsworker stdout | 2025-02-07 05:45:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:04 UTC)" (scheduled at 2025-02-07 05:45:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:45:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:45:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 4, 937386), True, datetime.datetime(2025, 2, 7, 5, 45, 4, 937386), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:45:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:45:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:45:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:45:05,711 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:45:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:45:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:18.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:45:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:18 UTC)" (scheduled at 2025-02-07 05:45:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:45:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:45:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 8, 732858), True, datetime.datetime(2025, 2, 7, 5, 45, 8, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:45:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:45:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:45:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:45:09,918 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:45:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:45:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:24.423055+00:00 (in 13.998335 seconds)
gcworker stdout | 2025-02-07 05:45:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:40 UTC)" (scheduled at 2025-02-07 05:45:10.424250+00:00)
gcworker stdout | 2025-02-07 05:45:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:45:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906810431, None, 1, 0])
gcworker stdout | 2025-02-07 05:45:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:45:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:45:11,028 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:45:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:45:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:41.831521+00:00 (in 29.999538 seconds)
autopruneworker stdout | 2025-02-07 05:45:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:41 UTC)" (scheduled at 2025-02-07 05:45:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:45:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903511836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:45:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:45:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:45:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:45:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:45:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:29.225378+00:00 (in 14.999531 seconds)
quotatotalworker stdout | 2025-02-07 05:45:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:29 UTC)" (scheduled at 2025-02-07 05:45:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:45:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:45:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:45:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:45:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:45:14,235 [238] [DEBUG] [app] Starting request: urn:request:0e975460-7b1c-4a5d-a844-dad0b58234fd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:14,235 [237] [DEBUG] [app] Starting request: urn:request:2e9b9ef5-3c35-46f1-afc7-9ee09fbf3da8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:45:14,238 [246] [DEBUG] [app] Starting request: urn:request:862194ca-3c03-4dd8-8dee-d9c5f68bfbb0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:14,238 [239] [DEBUG] [app] Starting request: urn:request:4cbb12fb-dba5-4fe7-a16b-779a53840b6d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:14,239 [246] [DEBUG] [app] Ending request: urn:request:862194ca-3c03-4dd8-8dee-d9c5f68bfbb0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:862194ca-3c03-4dd8-8dee-d9c5f68bfbb0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:45:14,239 [239] [DEBUG] [app] Ending request: urn:request:4cbb12fb-dba5-4fe7-a16b-779a53840b6d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4cbb12fb-dba5-4fe7-a16b-779a53840b6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:45:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:45:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:45:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:14,242 [235] [DEBUG] [app] Starting request: urn:request:0a4ab779-fadc-4d3d-9e51-8eed1beafc8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:14,242 [235] [DEBUG] [app] Ending request: urn:request:0a4ab779-fadc-4d3d-9e51-8eed1beafc8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a4ab779-fadc-4d3d-9e51-8eed1beafc8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:45:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:45:14,243 [237] [DEBUG] [app] Starting request: urn:request:7f6fcbc1-8a0b-4ba0-84e6-7323b6da495c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:14,244 [237] [DEBUG] [app] Ending request: urn:request:7f6fcbc1-8a0b-4ba0-84e6-7323b6da495c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f6fcbc1-8a0b-4ba0-84e6-7323b6da495c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:45:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:45:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:14,245 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:14,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:14,245 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:14,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:14,249 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:14,255 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:14,255 [237] [DEBUG] [app] Ending request: urn:request:2e9b9ef5-3c35-46f1-afc7-9ee09fbf3da8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e9b9ef5-3c35-46f1-afc7-9ee09fbf3da8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:45:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:45:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:45:14,256 [238] [DEBUG] [app] Ending request: urn:request:0e975460-7b1c-4a5d-a844-dad0b58234fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0e975460-7b1c-4a5d-a844-dad0b58234fd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:45:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:45:14,257 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.023)
quotatotalworker stdout | 2025-02-07 05:45:15,035 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:45:15,579 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:45:16,122 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:45:17,506 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:45:18,016 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:45:18,018 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:45:18,022 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:45:18,025 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:45:18,029 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:45:18,296 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:45:18,317 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:45:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:45:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:28.732147+00:00 (in 9.999569 seconds)
notificationworker stdout | 2025-02-07 05:45:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:28 UTC)" (scheduled at 2025-02-07 05:45:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:45:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:45:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 18, 732851), True, datetime.datetime(2025, 2, 7, 5, 45, 18, 732851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:45:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:45:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:45:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:45:19,033 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:45:19,036 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:45:19,038 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:45:19,681 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:19,888 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:45:20,307 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:45:20,466 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:45:20,501 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:45:20,507 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:45:20,512 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:45:20,515 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:45:20,525 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:45:20,528 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:45:20,531 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:45:20,538 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:45:20,541 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:45:20,676 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:45:21,012 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:45:21,433 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:45:21,694 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:45:21,770 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
repositorygcworker stdout | 2025-02-07 05:45:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:45:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:03.419110+00:00 (in 39.997765 seconds)
repositorygcworker stdout | 2025-02-07 05:45:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:23 UTC)" (scheduled at 2025-02-07 05:45:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:45:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:45:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:45:23,461 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:45:23,486 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:45:23,898 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:45:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:45:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:40.424250+00:00 (in 16.000730 seconds)
gcworker stdout | 2025-02-07 05:45:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:54 UTC)" (scheduled at 2025-02-07 05:45:24.423055+00:00)
gcworker stdout | 2025-02-07 05:45:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:45:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:45:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:45:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:45:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:38.732147+00:00 (in 9.999523 seconds)
notificationworker stdout | 2025-02-07 05:45:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:38 UTC)" (scheduled at 2025-02-07 05:45:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:45:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:45:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 28, 732924), True, datetime.datetime(2025, 2, 7, 5, 45, 28, 732924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:45:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:45:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:45:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:45:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:45:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:44.225378+00:00 (in 14.999513 seconds)
quotatotalworker stdout | 2025-02-07 05:45:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:44 UTC)" (scheduled at 2025-02-07 05:45:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:45:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:45:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:45:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:45:29,234 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:45:29,236 [235] [DEBUG] [app] Starting request: urn:request:f7bb4b7c-a72f-47f5-a0a5-90d570c932c4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:29,236 [238] [DEBUG] [app] Starting request: urn:request:6f89a78c-c6bc-4bf4-8b4c-38533604d930 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:45:29,239 [239] [DEBUG] [app] Starting request: urn:request:57b1d258-fd18-4b7f-9985-b9aa66cd1ce3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:29,239 [246] [DEBUG] [app] Starting request: urn:request:ffafbbfc-9e6e-4a98-9c90-4a54f98969e3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:29,239 [239] [DEBUG] [app] Ending request: urn:request:57b1d258-fd18-4b7f-9985-b9aa66cd1ce3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57b1d258-fd18-4b7f-9985-b9aa66cd1ce3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:45:29,239 [246] [DEBUG] [app] Ending request: urn:request:ffafbbfc-9e6e-4a98-9c90-4a54f98969e3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ffafbbfc-9e6e-4a98-9c90-4a54f98969e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:45:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:45:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:45:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:29,242 [237] [DEBUG] [app] Starting request: urn:request:4ed9beb5-e605-488b-b614-0e7c607cbd1d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:29,243 [237] [DEBUG] [app] Ending request: urn:request:4ed9beb5-e605-488b-b614-0e7c607cbd1d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4ed9beb5-e605-488b-b614-0e7c607cbd1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:45:29,243 [235] [DEBUG] [app] Starting request: urn:request:71fab695-fb29-4b2c-bcc2-b98b9f337e06 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:45:29,243 [235] [DEBUG] [app] Ending request: urn:request:71fab695-fb29-4b2c-bcc2-b98b9f337e06 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71fab695-fb29-4b2c-bcc2-b98b9f337e06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:45:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:45:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:29,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:29,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:29,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:29,255 [235] [DEBUG] [app] Ending request: urn:request:f7bb4b7c-a72f-47f5-a0a5-90d570c932c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f7bb4b7c-a72f-47f5-a0a5-90d570c932c4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:45:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:45:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:45:29,256 [238] [DEBUG] [app] Ending request: urn:request:6f89a78c-c6bc-4bf4-8b4c-38533604d930 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f89a78c-c6bc-4bf4-8b4c-38533604d930', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:45:29,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:45:29,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 05:45:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:45:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:32.625945+00:00 (in 1.004325 seconds)
namespacegcworker stdout | 2025-02-07 05:45:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:31 UTC)" (scheduled at 2025-02-07 05:45:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:45:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:45:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 31, 621922), True, datetime.datetime(2025, 2, 7, 5, 45, 31, 621922), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:45:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:45:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:45:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:45:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:45:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:31.621131+00:00 (in 58.994717 seconds)
namespacegcworker stdout | 2025-02-07 05:45:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:32 UTC)" (scheduled at 2025-02-07 05:45:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:45:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:45:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:45:32,902 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:45:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:45:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:03.919286+00:00 (in 29.999540 seconds)
buildlogsarchiver stdout | 2025-02-07 05:45:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:03 UTC)" (scheduled at 2025-02-07 05:45:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:45:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 45, 33, 919985), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:45:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:45:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:45:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:45:34,556 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:45:35,735 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:56.713547+00:00 (in 18.997142 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:37 UTC)" (scheduled at 2025-02-07 05:45:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:45:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:45:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:48.732147+00:00 (in 9.999501 seconds)
notificationworker stdout | 2025-02-07 05:45:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:48 UTC)" (scheduled at 2025-02-07 05:45:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:45:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:45:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 38, 732971), True, datetime.datetime(2025, 2, 7, 5, 45, 38, 732971), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:45:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:45:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:45:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:45:39,939 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:45:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:45:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:54.423055+00:00 (in 13.998351 seconds)
gcworker stdout | 2025-02-07 05:45:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:10 UTC)" (scheduled at 2025-02-07 05:45:40.424250+00:00)
gcworker stdout | 2025-02-07 05:45:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:45:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906840431, None, 1, 0])
gcworker stdout | 2025-02-07 05:45:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:45:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:45:41,058 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:45:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:45:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:11.831521+00:00 (in 29.999544 seconds)
autopruneworker stdout | 2025-02-07 05:45:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:11 UTC)" (scheduled at 2025-02-07 05:45:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:45:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903541836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:45:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:45:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:45:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:45:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:45:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:59.225378+00:00 (in 14.999561 seconds)
quotatotalworker stdout | 2025-02-07 05:45:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:59 UTC)" (scheduled at 2025-02-07 05:45:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:45:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:45:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:45:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:45:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:45:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:45:44,235 [238] [DEBUG] [app] Starting request: urn:request:51dbec69-66fa-4136-9b2c-98792829862c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:44,235 [237] [DEBUG] [app] Starting request: urn:request:20231ea0-cefc-439c-bb2d-26e67aa0dfff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:45:44,238 [239] [DEBUG] [app] Starting request: urn:request:13152d2e-ee4d-40b6-9c62-d7840dcb2b48 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:44,238 [246] [DEBUG] [app] Starting request: urn:request:00ce0559-05f9-43ef-95ba-6430712147be (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:44,238 [239] [DEBUG] [app] Ending request: urn:request:13152d2e-ee4d-40b6-9c62-d7840dcb2b48 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:13152d2e-ee4d-40b6-9c62-d7840dcb2b48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:45:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:45:44,239 [246] [DEBUG] [app] Ending request: urn:request:00ce0559-05f9-43ef-95ba-6430712147be (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:00ce0559-05f9-43ef-95ba-6430712147be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:45:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:45:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
gunicorn-web stdout | 2025-02-07 05:45:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:44,241 [236] [DEBUG] [app] Starting request: urn:request:bf99ab98-3461-4c4e-9870-e97eced9bee3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:44,242 [236] [DEBUG] [app] Ending request: urn:request:bf99ab98-3461-4c4e-9870-e97eced9bee3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf99ab98-3461-4c4e-9870-e97eced9bee3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:45:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:45:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:45:44,242 [235] [DEBUG] [app] Starting request: urn:request:a43546a2-f852-44e5-9184-617712242d55 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:44,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:44,243 [235] [DEBUG] [app] Ending request: urn:request:a43546a2-f852-44e5-9184-617712242d55 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a43546a2-f852-44e5-9184-617712242d55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:45:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:45:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:44,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:44,253 [237] [DEBUG] [app] Ending request: urn:request:20231ea0-cefc-439c-bb2d-26e67aa0dfff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20231ea0-cefc-439c-bb2d-26e67aa0dfff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:45:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:45:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:45:44,253 [238] [DEBUG] [app] Ending request: urn:request:51dbec69-66fa-4136-9b2c-98792829862c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:51dbec69-66fa-4136-9b2c-98792829862c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:45:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:45:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:45:45,071 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:45:45,609 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:45:46,129 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:45:47,542 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:45:48,023 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:45:48,026 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:45:48,029 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:45:48,032 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:45:48,035 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:45:48,331 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:45:48,352 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:45:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:45:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:50.814677+00:00 (in 2.082060 seconds)
notificationworker stdout | 2025-02-07 05:45:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:58 UTC)" (scheduled at 2025-02-07 05:45:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:45:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:45:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 48, 732874), True, datetime.datetime(2025, 2, 7, 5, 45, 48, 732874), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:45:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:45:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:45:48,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:45:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:45:49,041 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:45:49,044 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:45:49,046 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:45:49,718 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:49,906 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:45:50,343 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:45:50,502 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:45:50,507 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:45:50,517 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:45:50,521 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:45:50,525 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:45:50,533 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:45:50,537 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:45:50,540 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:45:50,545 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:45:50,548 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:45:50,687 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:45:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:45:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:45:58.732147+00:00 (in 7.917029 seconds)
notificationworker stdout | 2025-02-07 05:45:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:50 UTC)" (scheduled at 2025-02-07 05:45:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:45:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:45:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:50 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:45:51,049 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:45:51,460 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:45:51,730 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:45:51,777 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:45:53,497 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:45:53,511 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:45:53,934 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:45:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:45:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:10.424250+00:00 (in 16.000702 seconds)
gcworker stdout | 2025-02-07 05:45:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:24 UTC)" (scheduled at 2025-02-07 05:45:54.423055+00:00)
gcworker stdout | 2025-02-07 05:45:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:45:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:45:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:45:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:56.118855+00:00 (in 59.999500 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:45:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:56 UTC)" (scheduled at 2025-02-07 05:45:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:45:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:45:56,126 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:45:56,126 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:37.715878+00:00 (in 41.001822 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:56 UTC)" (scheduled at 2025-02-07 05:45:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:45:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:45:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:45:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:08.732147+00:00 (in 9.999538 seconds)
notificationworker stdout | 2025-02-07 05:45:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:08 UTC)" (scheduled at 2025-02-07 05:45:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:45:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:45:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 45, 58, 732904), True, datetime.datetime(2025, 2, 7, 5, 45, 58, 732904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:45:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:45:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:45:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:45:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:45:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:14.225378+00:00 (in 14.999582 seconds)
quotatotalworker stdout | 2025-02-07 05:45:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:14 UTC)" (scheduled at 2025-02-07 05:45:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:45:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:45:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:45:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:45:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:45:59,236 [237] [DEBUG] [app] Starting request: urn:request:440dd76d-983f-46a9-8de6-3277c366b165 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:59,236 [235] [DEBUG] [app] Starting request: urn:request:af588a06-8bc6-4ccd-8602-eeb4acbb9aad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:45:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:45:59,239 [242] [DEBUG] [app] Starting request: urn:request:ed809654-bf70-437c-8bd5-39b7af37f569 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:59,239 [239] [DEBUG] [app] Starting request: urn:request:4f0e64b5-dd83-4e1e-ac90-619d2ad40941 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:45:59,239 [239] [DEBUG] [app] Ending request: urn:request:4f0e64b5-dd83-4e1e-ac90-619d2ad40941 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f0e64b5-dd83-4e1e-ac90-619d2ad40941', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:45:59,239 [242] [DEBUG] [app] Ending request: urn:request:ed809654-bf70-437c-8bd5-39b7af37f569 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ed809654-bf70-437c-8bd5-39b7af37f569', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:45:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:45:59,240 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:45:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:45:59,242 [235] [DEBUG] [app] Starting request: urn:request:e7f54872-191c-4c8f-826d-5aa2c3c81513 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:59,242 [238] [DEBUG] [app] Starting request: urn:request:a01bd084-75e4-425b-a750-241f31b5e6b9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:45:59,243 [235] [DEBUG] [app] Ending request: urn:request:e7f54872-191c-4c8f-826d-5aa2c3c81513 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7f54872-191c-4c8f-826d-5aa2c3c81513', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:45:59,243 [238] [DEBUG] [app] Ending request: urn:request:a01bd084-75e4-425b-a750-241f31b5e6b9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a01bd084-75e4-425b-a750-241f31b5e6b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:45:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:45:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:45:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:45:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:45:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:45:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:45:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:45:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:45:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:45:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:45:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:59,255 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:45:59,255 [235] [DEBUG] [app] Ending request: urn:request:af588a06-8bc6-4ccd-8602-eeb4acbb9aad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af588a06-8bc6-4ccd-8602-eeb4acbb9aad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:45:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:45:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:45:59,256 [237] [DEBUG] [app] Ending request: urn:request:440dd76d-983f-46a9-8de6-3277c366b165 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:440dd76d-983f-46a9-8de6-3277c366b165', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:45:59,257 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:45:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
gunicorn-web stdout | 2025-02-07 05:45:59,257 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:45:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:45:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:45:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:04.936688+00:00 (in 4.998282 seconds)
exportactionlogsworker stdout | 2025-02-07 05:45:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:59 UTC)" (scheduled at 2025-02-07 05:45:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:45:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:45:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:46:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:46:02,937 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:46:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:46:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:23.420886+00:00 (in 20.001332 seconds)
repositorygcworker stdout | 2025-02-07 05:46:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:03 UTC)" (scheduled at 2025-02-07 05:46:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:46:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:46:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 3, 419851), True, datetime.datetime(2025, 2, 7, 5, 46, 3, 419851), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:46:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:46:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:46:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:46:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:46:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:33.919286+00:00 (in 29.999533 seconds)
buildlogsarchiver stdout | 2025-02-07 05:46:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:33 UTC)" (scheduled at 2025-02-07 05:46:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:46:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 46, 3, 920008), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:46:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:46:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:46:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:46:04,579 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:46:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:46:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:59.937899+00:00 (in 55.000693 seconds)
exportactionlogsworker stdout | 2025-02-07 05:46:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:04 UTC)" (scheduled at 2025-02-07 05:46:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:46:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:46:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 4, 937555), True, datetime.datetime(2025, 2, 7, 5, 46, 4, 937555), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:46:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:46:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:46:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:46:05,747 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:46:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:46:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:18.732147+00:00 (in 9.999557 seconds)
notificationworker stdout | 2025-02-07 05:46:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:18 UTC)" (scheduled at 2025-02-07 05:46:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:46:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:46:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 8, 732873), True, datetime.datetime(2025, 2, 7, 5, 46, 8, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:46:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:46:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:46:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:46:09,976 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:46:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:46:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:24.423055+00:00 (in 13.998360 seconds)
gcworker stdout | 2025-02-07 05:46:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:40 UTC)" (scheduled at 2025-02-07 05:46:10.424250+00:00)
gcworker stdout | 2025-02-07 05:46:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:46:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906870430, None, 1, 0])
gcworker stdout | 2025-02-07 05:46:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:46:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:46:11,095 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:46:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:46:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:41.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 05:46:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:41 UTC)" (scheduled at 2025-02-07 05:46:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:46:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903571836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:46:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:46:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:46:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:46:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:46:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:29.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 05:46:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:29 UTC)" (scheduled at 2025-02-07 05:46:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:46:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:46:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:46:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:46:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:46:14,235 [235] [DEBUG] [app] Starting request: urn:request:4687eb9c-4497-4130-81c0-d576421ba324 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:14,235 [238] [DEBUG] [app] Starting request: urn:request:7fb88beb-45ff-418c-8d28-6b96f73ca785 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:46:14,238 [239] [DEBUG] [app] Starting request: urn:request:555692e5-bba6-4729-ac04-f1cbd59eedc2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:14,238 [245] [DEBUG] [app] Starting request: urn:request:f2e3656b-de1f-47b5-86c0-86f473cd229f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:14,238 [239] [DEBUG] [app] Ending request: urn:request:555692e5-bba6-4729-ac04-f1cbd59eedc2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:555692e5-bba6-4729-ac04-f1cbd59eedc2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:46:14,239 [245] [DEBUG] [app] Ending request: urn:request:f2e3656b-de1f-47b5-86c0-86f473cd229f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f2e3656b-de1f-47b5-86c0-86f473cd229f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:46:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:46:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:14,242 [236] [DEBUG] [app] Starting request: urn:request:e5cb3de8-045e-4be5-bd5b-a02b45ae9e40 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:14,242 [237] [DEBUG] [app] Starting request: urn:request:3ad728b1-478b-49be-bc27-32faf4b34c4a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:14,242 [236] [DEBUG] [app] Ending request: urn:request:e5cb3de8-045e-4be5-bd5b-a02b45ae9e40 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e5cb3de8-045e-4be5-bd5b-a02b45ae9e40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:46:14,242 [237] [DEBUG] [app] Ending request: urn:request:3ad728b1-478b-49be-bc27-32faf4b34c4a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ad728b1-478b-49be-bc27-32faf4b34c4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:46:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:14,253 [238] [DEBUG] [app] Ending request: urn:request:7fb88beb-45ff-418c-8d28-6b96f73ca785 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7fb88beb-45ff-418c-8d28-6b96f73ca785', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:46:14,253 [235] [DEBUG] [app] Ending request: urn:request:4687eb9c-4497-4130-81c0-d576421ba324 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4687eb9c-4497-4130-81c0-d576421ba324', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:46:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:46:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:46:15,094 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:46:15,640 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:46:16,165 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:46:17,579 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:46:18,031 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:46:18,034 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:46:18,037 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:46:18,039 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:46:18,042 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:46:18,350 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:46:18,382 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:46:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:46:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:28.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 05:46:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:28 UTC)" (scheduled at 2025-02-07 05:46:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:46:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:46:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 18, 732877), True, datetime.datetime(2025, 2, 7, 5, 46, 18, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:46:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:46:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:46:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:46:19,048 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:46:19,051 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:46:19,054 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:46:19,754 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:19,939 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:46:20,379 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:46:20,519 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
globalpromstats stdout | 2025-02-07 05:46:20,524 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:46:20,528 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:46:20,531 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:46:20,535 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:46:20,540 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:46:20,545 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:46:20,549 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:46:20,552 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:46:20,555 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:46:20,717 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:46:21,086 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:46:21,497 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:46:21,767 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:46:21,813 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
repositorygcworker stdout | 2025-02-07 05:46:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:46:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:03.419110+00:00 (in 39.997773 seconds)
repositorygcworker stdout | 2025-02-07 05:46:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:23 UTC)" (scheduled at 2025-02-07 05:46:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:46:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:46:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:46:23,517 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:46:23,548 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:46:23,963 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:46:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:46:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:40.424250+00:00 (in 16.000707 seconds)
gcworker stdout | 2025-02-07 05:46:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:54 UTC)" (scheduled at 2025-02-07 05:46:24.423055+00:00)
gcworker stdout | 2025-02-07 05:46:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:46:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:46:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:46:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:46:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:38.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 05:46:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:38 UTC)" (scheduled at 2025-02-07 05:46:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:46:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:46:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 28, 732858), True, datetime.datetime(2025, 2, 7, 5, 46, 28, 732858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:46:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:46:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:46:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:46:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:46:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:44.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 05:46:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:44 UTC)" (scheduled at 2025-02-07 05:46:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:46:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:46:29,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:46:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:46:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:46:29,235 [237] [DEBUG] [app] Starting request: urn:request:e15639f8-585f-449b-a5fe-5fb57ac99676 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:29,235 [235] [DEBUG] [app] Starting request: urn:request:58a47bfb-f7c3-4cac-add0-0b7c0282149e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:46:29,238 [246] [DEBUG] [app] Starting request: urn:request:2b069758-8161-4997-a264-bdf2c7798797 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:29,238 [245] [DEBUG] [app] Starting request: urn:request:d08b75dc-4b8a-4145-bf36-e10fa5cec7d1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:29,238 [245] [DEBUG] [app] Ending request: urn:request:d08b75dc-4b8a-4145-bf36-e10fa5cec7d1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d08b75dc-4b8a-4145-bf36-e10fa5cec7d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:46:29,238 [246] [DEBUG] [app] Ending request: urn:request:2b069758-8161-4997-a264-bdf2c7798797 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2b069758-8161-4997-a264-bdf2c7798797', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:46:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:46:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:46:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:29,241 [237] [DEBUG] [app] Starting request: urn:request:6940fa8f-bfb4-41e2-a3b7-132483708b23 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:29,241 [238] [DEBUG] [app] Starting request: urn:request:6dd5078c-b136-4445-ae87-d3d91024e308 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:29,241 [237] [DEBUG] [app] Ending request: urn:request:6940fa8f-bfb4-41e2-a3b7-132483708b23 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6940fa8f-bfb4-41e2-a3b7-132483708b23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:46:29,241 [238] [DEBUG] [app] Ending request: urn:request:6dd5078c-b136-4445-ae87-d3d91024e308 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6dd5078c-b136-4445-ae87-d3d91024e308', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:46:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:46:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:29,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:29,253 [237] [DEBUG] [app] Ending request: urn:request:e15639f8-585f-449b-a5fe-5fb57ac99676 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e15639f8-585f-449b-a5fe-5fb57ac99676', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:46:29,253 [235] [DEBUG] [app] Ending request: urn:request:58a47bfb-f7c3-4cac-add0-0b7c0282149e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:58a47bfb-f7c3-4cac-add0-0b7c0282149e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:46:29,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:46:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:46:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:46:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:32.625945+00:00 (in 1.004353 seconds)
namespacegcworker stdout | 2025-02-07 05:46:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:31 UTC)" (scheduled at 2025-02-07 05:46:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:46:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:46:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 31, 621910), True, datetime.datetime(2025, 2, 7, 5, 46, 31, 621910), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:46:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:46:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:46:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:46:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:46:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:13.624301+00:00 (in 40.997892 seconds)
namespacegcworker stdout | 2025-02-07 05:46:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:32 UTC)" (scheduled at 2025-02-07 05:46:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:46:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:46:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:46:32,973 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:46:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:46:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:03.919286+00:00 (in 29.999564 seconds)
buildlogsarchiver stdout | 2025-02-07 05:46:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:03 UTC)" (scheduled at 2025-02-07 05:46:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:46:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 46, 33, 919975), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:46:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:46:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:46:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:46:34,592 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:46:35,769 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:56.713547+00:00 (in 18.997195 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:37 UTC)" (scheduled at 2025-02-07 05:46:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:46:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:46:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:48.732147+00:00 (in 9.999567 seconds)
notificationworker stdout | 2025-02-07 05:46:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:48 UTC)" (scheduled at 2025-02-07 05:46:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:46:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:46:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 38, 732843), True, datetime.datetime(2025, 2, 7, 5, 46, 38, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:46:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:46:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:46:38,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:46:40,013 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:46:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:46:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:54.423055+00:00 (in 13.998324 seconds)
gcworker stdout | 2025-02-07 05:46:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:10 UTC)" (scheduled at 2025-02-07 05:46:40.424250+00:00)
gcworker stdout | 2025-02-07 05:46:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:46:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906900431, None, 1, 0])
gcworker stdout | 2025-02-07 05:46:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:46:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:46:41,109 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:46:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:46:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:11.831521+00:00 (in 29.999516 seconds)
autopruneworker stdout | 2025-02-07 05:46:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:11 UTC)" (scheduled at 2025-02-07 05:46:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:46:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903601836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:46:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:46:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:46:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:46:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:46:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:59.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 05:46:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:59 UTC)" (scheduled at 2025-02-07 05:46:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:46:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:46:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:46:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:46:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:46:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:46:44,235 [236] [DEBUG] [app] Starting request: urn:request:c9d0bb43-89f0-43b4-b7c1-2cfa9b8399f3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:44,235 [237] [DEBUG] [app] Starting request: urn:request:118c80a8-221a-4a62-a899-b831df5f4aa1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:44,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:44,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:46:44,238 [246] [DEBUG] [app] Starting request: urn:request:6d297507-b1aa-45ee-b1e8-519fa0970f72 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:44,238 [239] [DEBUG] [app] Starting request: urn:request:ef3dd6d7-c474-45b5-bac9-46a24ec28413 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:44,238 [239] [DEBUG] [app] Ending request: urn:request:ef3dd6d7-c474-45b5-bac9-46a24ec28413 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef3dd6d7-c474-45b5-bac9-46a24ec28413', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:46:44,238 [246] [DEBUG] [app] Ending request: urn:request:6d297507-b1aa-45ee-b1e8-519fa0970f72 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d297507-b1aa-45ee-b1e8-519fa0970f72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:46:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:46:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:44,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:44,241 [235] [DEBUG] [app] Starting request: urn:request:3c82fdc9-88f8-42e4-93d8-ab4348bd3569 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:44,241 [237] [DEBUG] [app] Starting request: urn:request:c030df68-b599-4a44-b866-e71983705bfe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:44,242 [237] [DEBUG] [app] Ending request: urn:request:c030df68-b599-4a44-b866-e71983705bfe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c030df68-b599-4a44-b866-e71983705bfe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:46:44,242 [235] [DEBUG] [app] Ending request: urn:request:3c82fdc9-88f8-42e4-93d8-ab4348bd3569 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c82fdc9-88f8-42e4-93d8-ab4348bd3569', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:46:44,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:44,242 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:44,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:44,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:44,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:44,246 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:44,246 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:44,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:44,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:44,250 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:44,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:44,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:44,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:44,253 [237] [DEBUG] [app] Ending request: urn:request:118c80a8-221a-4a62-a899-b831df5f4aa1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:118c80a8-221a-4a62-a899-b831df5f4aa1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:44,253 [236] [DEBUG] [app] Ending request: urn:request:c9d0bb43-89f0-43b4-b7c1-2cfa9b8399f3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c9d0bb43-89f0-43b4-b7c1-2cfa9b8399f3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:44,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:46:44,253 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:46:44,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:46:44,253 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:46:45,129 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:46:45,675 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:46:46,193 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:46:47,610 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:46:48,040 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:46:48,043 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:46:48,046 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:46:48,049 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:46:48,052 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:46:48,371 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:46:48,418 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:46:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:46:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:50.814677+00:00 (in 2.082069 seconds)
notificationworker stdout | 2025-02-07 05:46:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:58 UTC)" (scheduled at 2025-02-07 05:46:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:46:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:46:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 48, 732910), True, datetime.datetime(2025, 2, 7, 5, 46, 48, 732910), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:46:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:46:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:46:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:46:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:46:49,057 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:46:49,060 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:46:49,063 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:46:49,790 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:49,967 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:46:50,407 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:46:50,531 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:46:50,537 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:46:50,540 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:46:50,543 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:46:50,550 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:46:50,553 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:46:50,557 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
globalpromstats stdout | 2025-02-07 05:46:50,560 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
gunicorn-registry stdout | 2025-02-07 05:46:50,562 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:46:50,564 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
exportactionlogsworker stdout | 2025-02-07 05:46:50,735 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:46:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:46:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:46:58.732147+00:00 (in 7.916940 seconds)
notificationworker stdout | 2025-02-07 05:46:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:50 UTC)" (scheduled at 2025-02-07 05:46:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:46:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:46:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:50 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:46:51,113 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:46:51,533 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:46:51,803 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:46:51,843 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:46:53,553 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:46:53,584 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:46:54,000 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:46:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:46:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:10.424250+00:00 (in 16.000742 seconds)
gcworker stdout | 2025-02-07 05:46:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:24 UTC)" (scheduled at 2025-02-07 05:46:54.423055+00:00)
gcworker stdout | 2025-02-07 05:46:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:46:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:46:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:46:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:56.118855+00:00 (in 59.999504 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:46:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:56 UTC)" (scheduled at 2025-02-07 05:46:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:46:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:46:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:46:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:37.715878+00:00 (in 41.001847 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:56 UTC)" (scheduled at 2025-02-07 05:46:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:46:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:46:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:46:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:08.732147+00:00 (in 9.999469 seconds)
notificationworker stdout | 2025-02-07 05:46:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:08 UTC)" (scheduled at 2025-02-07 05:46:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:46:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:46:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 46, 58, 732981), True, datetime.datetime(2025, 2, 7, 5, 46, 58, 732981), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:46:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:46:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:46:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:46:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:46:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:14.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 05:46:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:14 UTC)" (scheduled at 2025-02-07 05:46:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:46:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:46:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:46:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:46:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:46:59,236 [236] [DEBUG] [app] Starting request: urn:request:11e6c6fc-eea1-41b3-b48a-51b4868e46b8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:59,236 [235] [DEBUG] [app] Starting request: urn:request:dbf1b126-47fa-48c2-b6bb-e073a98b5556 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:46:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:46:59,238 [239] [DEBUG] [app] Starting request: urn:request:b1022624-e635-4247-b7e8-ea99d85fb9f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:59,239 [246] [DEBUG] [app] Starting request: urn:request:c53b8817-2d5f-4021-8b73-204306438be5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:46:59,239 [239] [DEBUG] [app] Ending request: urn:request:b1022624-e635-4247-b7e8-ea99d85fb9f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1022624-e635-4247-b7e8-ea99d85fb9f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:46:59,239 [246] [DEBUG] [app] Ending request: urn:request:c53b8817-2d5f-4021-8b73-204306438be5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c53b8817-2d5f-4021-8b73-204306438be5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:46:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:46:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:46:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:46:59,242 [237] [DEBUG] [app] Starting request: urn:request:bf161b9d-7297-43e0-839e-570e6d9f777c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:59,242 [237] [DEBUG] [app] Ending request: urn:request:bf161b9d-7297-43e0-839e-570e6d9f777c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bf161b9d-7297-43e0-839e-570e6d9f777c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:46:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:46:59,242 [236] [DEBUG] [app] Starting request: urn:request:14762fe4-6525-4477-ae0a-c461f80f52ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:46:59,243 [236] [DEBUG] [app] Ending request: urn:request:14762fe4-6525-4477-ae0a-c461f80f52ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:14762fe4-6525-4477-ae0a-c461f80f52ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:46:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:46:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:46:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:46:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:46:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:46:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:46:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:46:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:46:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:46:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:46:59,254 [235] [DEBUG] [app] Ending request: urn:request:dbf1b126-47fa-48c2-b6bb-e073a98b5556 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dbf1b126-47fa-48c2-b6bb-e073a98b5556', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:46:59,254 [236] [DEBUG] [app] Ending request: urn:request:11e6c6fc-eea1-41b3-b48a-51b4868e46b8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11e6c6fc-eea1-41b3-b48a-51b4868e46b8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:46:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:46:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:46:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
gunicorn-web stdout | 2025-02-07 05:46:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:46:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:46:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:46:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:04.936688+00:00 (in 4.998267 seconds)
exportactionlogsworker stdout | 2025-02-07 05:46:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:59 UTC)" (scheduled at 2025-02-07 05:46:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:46:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:46:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:47:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:47:03,009 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:47:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:47:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:23.420886+00:00 (in 20.001342 seconds)
repositorygcworker stdout | 2025-02-07 05:47:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:03 UTC)" (scheduled at 2025-02-07 05:47:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:47:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:47:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 3, 419815), True, datetime.datetime(2025, 2, 7, 5, 47, 3, 419815), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:47:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:47:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:47:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:47:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:47:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:33.919286+00:00 (in 29.999555 seconds)
buildlogsarchiver stdout | 2025-02-07 05:47:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:33 UTC)" (scheduled at 2025-02-07 05:47:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:47:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 47, 3, 919970), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:47:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:47:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:47:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:47:04,618 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:47:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:47:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:59.937899+00:00 (in 55.000777 seconds)
exportactionlogsworker stdout | 2025-02-07 05:47:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:04 UTC)" (scheduled at 2025-02-07 05:47:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:47:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:47:04,937 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 4, 937351), True, datetime.datetime(2025, 2, 7, 5, 47, 4, 937351), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:47:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:47:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:47:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:47:05,806 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:47:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:18.732147+00:00 (in 9.999493 seconds)
notificationworker stdout | 2025-02-07 05:47:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:18 UTC)" (scheduled at 2025-02-07 05:47:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:47:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:47:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 8, 732952), True, datetime.datetime(2025, 2, 7, 5, 47, 8, 732952), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:47:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:47:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:47:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:47:10,047 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:47:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:47:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:24.423055+00:00 (in 13.998362 seconds)
gcworker stdout | 2025-02-07 05:47:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:40 UTC)" (scheduled at 2025-02-07 05:47:10.424250+00:00)
gcworker stdout | 2025-02-07 05:47:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:47:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906930431, None, 1, 0])
gcworker stdout | 2025-02-07 05:47:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:47:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:47:11,141 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:47:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:47:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:41.831521+00:00 (in 29.999557 seconds)
autopruneworker stdout | 2025-02-07 05:47:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:41 UTC)" (scheduled at 2025-02-07 05:47:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:47:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903631836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:47:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:47:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:47:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:41 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:47:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:47:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:31.621131+00:00 (in 17.996395 seconds)
namespacegcworker stdout | 2025-02-07 05:47:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:52:13 UTC)" (scheduled at 2025-02-07 05:47:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:47:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 47, 13, 624971), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:47:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 47, 13, 624971), True, datetime.datetime(2025, 2, 7, 5, 47, 13, 624971), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:47:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 47, 13, 624971), True, datetime.datetime(2025, 2, 7, 5, 47, 13, 624971), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 47, 13, 624971), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:47:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:47:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:52:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:47:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:47:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:29.225378+00:00 (in 14.999562 seconds)
quotatotalworker stdout | 2025-02-07 05:47:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:29 UTC)" (scheduled at 2025-02-07 05:47:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:47:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:47:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:47:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:47:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:47:14,235 [235] [DEBUG] [app] Starting request: urn:request:6dbf87dd-577d-45fd-acda-bb4c55568fd8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:14,235 [238] [DEBUG] [app] Starting request: urn:request:1e2ee3bd-beba-4fd0-a74c-98bb4811f9e3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:14,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:47:14,238 [239] [DEBUG] [app] Starting request: urn:request:0d6f015a-293e-4c8d-a4eb-e525e3be431b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:14,238 [245] [DEBUG] [app] Starting request: urn:request:749bc17d-d9b1-4254-9aaf-224da414df72 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:14,238 [239] [DEBUG] [app] Ending request: urn:request:0d6f015a-293e-4c8d-a4eb-e525e3be431b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d6f015a-293e-4c8d-a4eb-e525e3be431b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:47:14,238 [245] [DEBUG] [app] Ending request: urn:request:749bc17d-d9b1-4254-9aaf-224da414df72 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:749bc17d-d9b1-4254-9aaf-224da414df72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:47:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:47:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:14,241 [238] [DEBUG] [app] Starting request: urn:request:e3520751-9b7c-4e0e-93b2-d48e06ae519b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:47:14,241 [236] [DEBUG] [app] Starting request: urn:request:e05e89c9-76cc-454e-8e4f-b66b58b661cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:47:14,241 [238] [DEBUG] [app] Ending request: urn:request:e3520751-9b7c-4e0e-93b2-d48e06ae519b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e3520751-9b7c-4e0e-93b2-d48e06ae519b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:47:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:14,242 [236] [DEBUG] [app] Ending request: urn:request:e05e89c9-76cc-454e-8e4f-b66b58b661cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e05e89c9-76cc-454e-8e4f-b66b58b661cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:47:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:47:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:47:14,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:47:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:14,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:14,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:14,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:14,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:14,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:14,252 [238] [DEBUG] [app] Ending request: urn:request:1e2ee3bd-beba-4fd0-a74c-98bb4811f9e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e2ee3bd-beba-4fd0-a74c-98bb4811f9e3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:14,253 [235] [DEBUG] [app] Ending request: urn:request:6dbf87dd-577d-45fd-acda-bb4c55568fd8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6dbf87dd-577d-45fd-acda-bb4c55568fd8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:47:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:47:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:47:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
quotatotalworker stdout | 2025-02-07 05:47:15,165 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:47:15,711 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:47:16,229 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:47:17,646 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:47:18,047 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:47:18,050 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:47:18,053 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:47:18,056 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:47:18,059 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:47:18,401 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:47:18,454 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:47:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:28.732147+00:00 (in 9.999559 seconds)
notificationworker stdout | 2025-02-07 05:47:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:28 UTC)" (scheduled at 2025-02-07 05:47:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:47:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:47:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 18, 732863), True, datetime.datetime(2025, 2, 7, 5, 47, 18, 732863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:47:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:47:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:47:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:47:19,064 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:47:19,067 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:47:19,071 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:47:19,817 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:20,003 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:47:20,435 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:47:20,543 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:47:20,546 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:47:20,550 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:47:20,552 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:47:20,556 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:47:20,559 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:47:20,565 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:47:20,568 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:47:20,572 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:47:20,576 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:47:20,765 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:47:21,140 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:47:21,569 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:47:21,839 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:47:21,878 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
repositorygcworker stdout | 2025-02-07 05:47:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:47:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:03.419110+00:00 (in 39.997758 seconds)
repositorygcworker stdout | 2025-02-07 05:47:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:23 UTC)" (scheduled at 2025-02-07 05:47:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:47:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:47:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:47:23,590 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:47:23,619 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:47:24,036 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:47:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:47:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:40.424250+00:00 (in 16.000729 seconds)
gcworker stdout | 2025-02-07 05:47:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:54 UTC)" (scheduled at 2025-02-07 05:47:24.423055+00:00)
gcworker stdout | 2025-02-07 05:47:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:47:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:47:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:47:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:33.814166+00:00 (in 5.081582 seconds)
notificationworker stdout | 2025-02-07 05:47:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:38 UTC)" (scheduled at 2025-02-07 05:47:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:47:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:47:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 28, 732869), True, datetime.datetime(2025, 2, 7, 5, 47, 28, 732869), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:47:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:47:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:47:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:47:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:47:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:44.225378+00:00 (in 14.999529 seconds)
quotatotalworker stdout | 2025-02-07 05:47:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:44 UTC)" (scheduled at 2025-02-07 05:47:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:47:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:47:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:47:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:47:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:47:29,236 [238] [DEBUG] [app] Starting request: urn:request:639a4df3-a3a7-4100-b5ca-3b3dca8ea671 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:29,236 [236] [DEBUG] [app] Starting request: urn:request:9c35fb2b-b44c-4774-8998-69edb36fa12c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:29,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:29,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:47:29,239 [245] [DEBUG] [app] Starting request: urn:request:fce27c70-af10-4407-ae64-2d12e371b262 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:29,239 [239] [DEBUG] [app] Starting request: urn:request:e7e09f52-9d8b-4f6f-88a5-d1b44680d9ac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:29,240 [239] [DEBUG] [app] Ending request: urn:request:e7e09f52-9d8b-4f6f-88a5-d1b44680d9ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e7e09f52-9d8b-4f6f-88a5-d1b44680d9ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:47:29,240 [245] [DEBUG] [app] Ending request: urn:request:fce27c70-af10-4407-ae64-2d12e371b262 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fce27c70-af10-4407-ae64-2d12e371b262', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:47:29,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:47:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:29,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:47:29,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:29,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:29,242 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:29,242 [236] [DEBUG] [app] Starting request: urn:request:9982d050-3bf6-405e-b30d-85e3a7c3d055 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:47:29,243 [236] [DEBUG] [app] Ending request: urn:request:9982d050-3bf6-405e-b30d-85e3a7c3d055 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9982d050-3bf6-405e-b30d-85e3a7c3d055', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:47:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:29,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:29,243 [235] [DEBUG] [app] Starting request: urn:request:92108fcc-4efe-4d63-a34b-4842a8764453 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:47:29,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:29,244 [235] [DEBUG] [app] Ending request: urn:request:92108fcc-4efe-4d63-a34b-4842a8764453 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:92108fcc-4efe-4d63-a34b-4842a8764453', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:47:29,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:47:29,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:29,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:29,245 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:29,245 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:47:29,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:29,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:29,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:29,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:29,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:29,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:29,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:29,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:29,255 [236] [DEBUG] [app] Ending request: urn:request:9c35fb2b-b44c-4774-8998-69edb36fa12c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9c35fb2b-b44c-4774-8998-69edb36fa12c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:29,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:47:29,255 [238] [DEBUG] [app] Ending request: urn:request:639a4df3-a3a7-4100-b5ca-3b3dca8ea671 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:639a4df3-a3a7-4100-b5ca-3b3dca8ea671', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:29,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:47:29,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:47:29,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 05:47:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:47:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:32.625945+00:00 (in 1.004382 seconds)
namespacegcworker stdout | 2025-02-07 05:47:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:31 UTC)" (scheduled at 2025-02-07 05:47:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:47:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:47:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 31, 621872), True, datetime.datetime(2025, 2, 7, 5, 47, 31, 621872), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:47:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:47:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:47:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:47:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:47:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:31.621131+00:00 (in 58.994695 seconds)
namespacegcworker stdout | 2025-02-07 05:47:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:32 UTC)" (scheduled at 2025-02-07 05:47:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:47:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:47:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:47:33,033 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 05:47:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:38.732147+00:00 (in 4.917546 seconds)
notificationworker stdout | 2025-02-07 05:47:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:52:33 UTC)" (scheduled at 2025-02-07 05:47:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:47:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 47, 33, 814853), 'notification/%'])
notificationworker stdout | 2025-02-07 05:47:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 47, 33, 814853), True, datetime.datetime(2025, 2, 7, 5, 47, 33, 814853), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:47:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 47, 33, 814853), True, datetime.datetime(2025, 2, 7, 5, 47, 33, 814853), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 47, 33, 814853), 'notification/%'])
notificationworker stdout | 2025-02-07 05:47:33,824 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:47:33,824 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:52:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:47:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:47:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:03.919286+00:00 (in 29.999585 seconds)
buildlogsarchiver stdout | 2025-02-07 05:47:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:03 UTC)" (scheduled at 2025-02-07 05:47:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:47:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 47, 33, 919942), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:47:33,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:47:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:47:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:47:34,654 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:47:35,816 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:56.713547+00:00 (in 18.997161 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:37 UTC)" (scheduled at 2025-02-07 05:47:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:47:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:48.732147+00:00 (in 9.999615 seconds)
notificationworker stdout | 2025-02-07 05:47:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:48 UTC)" (scheduled at 2025-02-07 05:47:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:47:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:47:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 38, 732821), True, datetime.datetime(2025, 2, 7, 5, 47, 38, 732821), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:47:38,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:47:38,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:47:38,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:47:40,083 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:47:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:47:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:54.423055+00:00 (in 13.998381 seconds)
gcworker stdout | 2025-02-07 05:47:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:10 UTC)" (scheduled at 2025-02-07 05:47:40.424250+00:00)
gcworker stdout | 2025-02-07 05:47:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:47:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906960430, None, 1, 0])
gcworker stdout | 2025-02-07 05:47:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:47:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:47:41,178 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:47:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:47:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:11.831521+00:00 (in 29.999586 seconds)
autopruneworker stdout | 2025-02-07 05:47:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:11 UTC)" (scheduled at 2025-02-07 05:47:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:47:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903661836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:47:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:47:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:47:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:47:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:47:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:59.225378+00:00 (in 14.999512 seconds)
quotatotalworker stdout | 2025-02-07 05:47:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:59 UTC)" (scheduled at 2025-02-07 05:47:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:47:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:47:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:47:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:47:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:47:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:47:44,235 [235] [DEBUG] [app] Starting request: urn:request:16b63229-c399-4185-9e7d-cf99c639ac10 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:44,235 [236] [DEBUG] [app] Starting request: urn:request:3b1c3706-6afa-4db0-97a9-fbad318c5b98 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:47:44,238 [246] [DEBUG] [app] Starting request: urn:request:1e9cf32e-1913-4958-89f5-bcbc6239c019 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:44,238 [239] [DEBUG] [app] Starting request: urn:request:d5f94cc5-58d4-47cd-a35e-09f2cce4b222 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:44,239 [246] [DEBUG] [app] Ending request: urn:request:1e9cf32e-1913-4958-89f5-bcbc6239c019 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e9cf32e-1913-4958-89f5-bcbc6239c019', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:47:44,239 [239] [DEBUG] [app] Ending request: urn:request:d5f94cc5-58d4-47cd-a35e-09f2cce4b222 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d5f94cc5-58d4-47cd-a35e-09f2cce4b222', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:47:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:47:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:47:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:44,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:44,241 [237] [DEBUG] [app] Starting request: urn:request:d0db5dee-d69b-47e4-a330-97555b1e62d8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:47:44,242 [237] [DEBUG] [app] Ending request: urn:request:d0db5dee-d69b-47e4-a330-97555b1e62d8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d0db5dee-d69b-47e4-a330-97555b1e62d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:47:44,242 [236] [DEBUG] [app] Starting request: urn:request:5074857d-b3c0-4bc4-a909-3882b8d2999e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:47:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:44,242 [236] [DEBUG] [app] Ending request: urn:request:5074857d-b3c0-4bc4-a909-3882b8d2999e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5074857d-b3c0-4bc4-a909-3882b8d2999e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.000 159 0.001)
gunicorn-web stdout | 2025-02-07 05:47:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:44,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:44,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:47:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:47:44,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:44,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:44,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:44,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:44,254 [235] [DEBUG] [app] Ending request: urn:request:16b63229-c399-4185-9e7d-cf99c639ac10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16b63229-c399-4185-9e7d-cf99c639ac10', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:44,254 [236] [DEBUG] [app] Ending request: urn:request:3b1c3706-6afa-4db0-97a9-fbad318c5b98 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3b1c3706-6afa-4db0-97a9-fbad318c5b98', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:47:44,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:47:44,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:47:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
quotatotalworker stdout | 2025-02-07 05:47:45,202 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:47:45,730 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:47:46,266 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:47:47,682 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:47:48,055 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:47:48,058 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:47:48,061 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:47:48,065 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:47:48,067 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:47:48,431 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:47:48,465 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:47:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:50.814677+00:00 (in 2.082100 seconds)
notificationworker stdout | 2025-02-07 05:47:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:58 UTC)" (scheduled at 2025-02-07 05:47:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:47:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:47:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 48, 732844), True, datetime.datetime(2025, 2, 7, 5, 47, 48, 732844), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:47:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:47:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:47:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:47:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:47:49,072 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:47:49,074 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:47:49,077 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:47:49,831 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:50,012 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:47:50,442 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:47:50,554 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:47:50,557 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:47:50,560 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:47:50,563 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:47:50,566 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:47:50,569 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:47:50,572 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:47:50,581 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:47:50,584 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:47:50,590 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:47:50,774 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
notificationworker stdout | 2025-02-07 05:47:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:47:58.732147+00:00 (in 7.916989 seconds)
notificationworker stdout | 2025-02-07 05:47:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:50 UTC)" (scheduled at 2025-02-07 05:47:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:47:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:47:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:50 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:47:51,177 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:47:51,578 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:47:51,876 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:47:51,914 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:47:53,626 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:47:53,649 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:47:54,072 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:47:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:47:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:10.424250+00:00 (in 16.000741 seconds)
gcworker stdout | 2025-02-07 05:47:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:24 UTC)" (scheduled at 2025-02-07 05:47:54.423055+00:00)
gcworker stdout | 2025-02-07 05:47:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:47:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:47:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:47:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:56.118855+00:00 (in 59.999514 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:47:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:56 UTC)" (scheduled at 2025-02-07 05:47:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:47:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:47:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:47:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:37.715878+00:00 (in 41.001847 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:56 UTC)" (scheduled at 2025-02-07 05:47:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:47:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:47:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:47:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:08.732147+00:00 (in 9.999504 seconds)
notificationworker stdout | 2025-02-07 05:47:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:08 UTC)" (scheduled at 2025-02-07 05:47:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:47:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:47:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 47, 58, 732974), True, datetime.datetime(2025, 2, 7, 5, 47, 58, 732974), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:47:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:47:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:47:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:47:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:47:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:14.225378+00:00 (in 14.999507 seconds)
quotatotalworker stdout | 2025-02-07 05:47:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:14 UTC)" (scheduled at 2025-02-07 05:47:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:47:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:47:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:47:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:47:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:47:59,236 [235] [DEBUG] [app] Starting request: urn:request:4509e793-6ebb-410b-b5f3-9e39cf4eacd4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:59,236 [238] [DEBUG] [app] Starting request: urn:request:31247645-da8a-4956-ae4a-0aa08e693f5e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:47:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:47:59,239 [239] [DEBUG] [app] Starting request: urn:request:5defb754-d512-4c7a-85c3-ecf74041461d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:59,239 [245] [DEBUG] [app] Starting request: urn:request:b83f9d2c-12d1-4149-b974-ccc042d4116e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:47:59,239 [239] [DEBUG] [app] Ending request: urn:request:5defb754-d512-4c7a-85c3-ecf74041461d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5defb754-d512-4c7a-85c3-ecf74041461d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:47:59,239 [245] [DEBUG] [app] Ending request: urn:request:b83f9d2c-12d1-4149-b974-ccc042d4116e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b83f9d2c-12d1-4149-b974-ccc042d4116e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:47:59,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:47:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:59,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:47:59,242 [238] [DEBUG] [app] Starting request: urn:request:a4d96efa-9dcc-46fc-ab01-1e6143069d9e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:47:59,242 [236] [DEBUG] [app] Starting request: urn:request:f786501d-adcc-4690-99d1-714c1f7e31f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:47:59,243 [238] [DEBUG] [app] Ending request: urn:request:a4d96efa-9dcc-46fc-ab01-1e6143069d9e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4d96efa-9dcc-46fc-ab01-1e6143069d9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:47:59,243 [236] [DEBUG] [app] Ending request: urn:request:f786501d-adcc-4690-99d1-714c1f7e31f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f786501d-adcc-4690-99d1-714c1f7e31f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:47:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:47:59,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:47:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:47:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:47:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:47:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:47:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:47:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:47:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:47:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:47:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:47:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:59,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:47:59,255 [235] [DEBUG] [app] Ending request: urn:request:4509e793-6ebb-410b-b5f3-9e39cf4eacd4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4509e793-6ebb-410b-b5f3-9e39cf4eacd4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:59,255 [238] [DEBUG] [app] Ending request: urn:request:31247645-da8a-4956-ae4a-0aa08e693f5e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:31247645-da8a-4956-ae4a-0aa08e693f5e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:47:59,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:47:59,256 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:47:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:47:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:47:59,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:47:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:47:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:47:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:04.936688+00:00 (in 4.998302 seconds)
exportactionlogsworker stdout | 2025-02-07 05:47:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:59 UTC)" (scheduled at 2025-02-07 05:47:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:47:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:47:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:48:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:48:03,069 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:48:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:48:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:23.420886+00:00 (in 20.001337 seconds)
repositorygcworker stdout | 2025-02-07 05:48:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:03 UTC)" (scheduled at 2025-02-07 05:48:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:48:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:48:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 3, 419816), True, datetime.datetime(2025, 2, 7, 5, 48, 3, 419816), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:48:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:48:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:48:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:48:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:48:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:33.919286+00:00 (in 29.999588 seconds)
buildlogsarchiver stdout | 2025-02-07 05:48:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:33 UTC)" (scheduled at 2025-02-07 05:48:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:48:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 48, 3, 919949), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:48:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:48:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:48:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:48:04,691 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:48:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:48:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:59.937899+00:00 (in 55.000765 seconds)
exportactionlogsworker stdout | 2025-02-07 05:48:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:04 UTC)" (scheduled at 2025-02-07 05:48:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:48:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:48:04,937 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 4, 937355), True, datetime.datetime(2025, 2, 7, 5, 48, 4, 937355), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:48:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:48:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:48:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:48:05,841 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:48:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:48:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:18.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 05:48:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:18 UTC)" (scheduled at 2025-02-07 05:48:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:48:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:48:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 8, 732859), True, datetime.datetime(2025, 2, 7, 5, 48, 8, 732859), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:48:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:48:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:48:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:48:10,108 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:48:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:48:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:24.423055+00:00 (in 13.998340 seconds)
gcworker stdout | 2025-02-07 05:48:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:40 UTC)" (scheduled at 2025-02-07 05:48:10.424250+00:00)
gcworker stdout | 2025-02-07 05:48:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:48:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738906990431, None, 1, 0])
gcworker stdout | 2025-02-07 05:48:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:48:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:48:11,206 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:48:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:48:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:41.831521+00:00 (in 29.999556 seconds)
autopruneworker stdout | 2025-02-07 05:48:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:41 UTC)" (scheduled at 2025-02-07 05:48:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:48:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903691836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:48:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:48:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:48:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:48:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:48:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:29.225378+00:00 (in 14.999551 seconds)
quotatotalworker stdout | 2025-02-07 05:48:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:29 UTC)" (scheduled at 2025-02-07 05:48:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:48:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:48:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:48:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:48:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:48:14,235 [238] [DEBUG] [app] Starting request: urn:request:8bc01634-ae64-4391-87ac-23b88360d7cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:14,235 [237] [DEBUG] [app] Starting request: urn:request:ea601800-d072-4f2e-84bc-86dac47c6161 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:48:14,238 [246] [DEBUG] [app] Starting request: urn:request:720cec71-fd82-41e9-a071-ea41849f2bad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:14,238 [239] [DEBUG] [app] Starting request: urn:request:613354df-501b-45ac-9da1-ded0b7f41c0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:14,238 [239] [DEBUG] [app] Ending request: urn:request:613354df-501b-45ac-9da1-ded0b7f41c0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:613354df-501b-45ac-9da1-ded0b7f41c0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:48:14,238 [246] [DEBUG] [app] Ending request: urn:request:720cec71-fd82-41e9-a071-ea41849f2bad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:720cec71-fd82-41e9-a071-ea41849f2bad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:48:14,238 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:48:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:48:14,238 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:48:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:14,241 [237] [DEBUG] [app] Starting request: urn:request:43da4eb9-fc2d-4954-89a0-f4c408b631fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:14,241 [235] [DEBUG] [app] Starting request: urn:request:27b63a9a-07ca-42cb-9eb0-12a6a6125d39 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:14,241 [237] [DEBUG] [app] Ending request: urn:request:43da4eb9-fc2d-4954-89a0-f4c408b631fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:43da4eb9-fc2d-4954-89a0-f4c408b631fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:48:14,242 [235] [DEBUG] [app] Ending request: urn:request:27b63a9a-07ca-42cb-9eb0-12a6a6125d39 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27b63a9a-07ca-42cb-9eb0-12a6a6125d39', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:48:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:48:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:48:14,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:48:14,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:14,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:14,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:14,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:14,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:14,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:14,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:14,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:14,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:14,253 [237] [DEBUG] [app] Ending request: urn:request:ea601800-d072-4f2e-84bc-86dac47c6161 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ea601800-d072-4f2e-84bc-86dac47c6161', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:48:14,253 [238] [DEBUG] [app] Ending request: urn:request:8bc01634-ae64-4391-87ac-23b88360d7cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8bc01634-ae64-4391-87ac-23b88360d7cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:48:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:48:14,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:48:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:48:14,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:48:15,238 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:48:15,743 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:48:16,302 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:48:17,718 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:48:18,063 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:48:18,066 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:48:18,069 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:48:18,071 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:48:18,074 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:48:18,439 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:48:18,501 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:48:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:48:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:28.732147+00:00 (in 9.999544 seconds)
notificationworker stdout | 2025-02-07 05:48:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:28 UTC)" (scheduled at 2025-02-07 05:48:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:48:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:48:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 18, 732817), True, datetime.datetime(2025, 2, 7, 5, 48, 18, 732817), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:48:18,757 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:48:18,757 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:48:18,757 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:48:19,079 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:48:19,082 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:48:19,084 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:48:19,843 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:20,026 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:48:20,474 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:48:20,564 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:48:20,566 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:48:20,569 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:48:20,572 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:48:20,578 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:48:20,581 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:48:20,583 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:48:20,592 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:48:20,595 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:48:20,618 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:48:20,810 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:48:21,213 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:48:21,592 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:48:21,901 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:48:21,924 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
repositorygcworker stdout | 2025-02-07 05:48:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:48:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:32.420707+00:00 (in 8.999355 seconds)
repositorygcworker stdout | 2025-02-07 05:48:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:23 UTC)" (scheduled at 2025-02-07 05:48:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:48:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:48:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:48:23,640 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:48:23,685 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:48:24,088 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:48:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:48:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:40.424250+00:00 (in 16.000713 seconds)
gcworker stdout | 2025-02-07 05:48:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:54 UTC)" (scheduled at 2025-02-07 05:48:24.423055+00:00)
gcworker stdout | 2025-02-07 05:48:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:48:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:48:54 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:48:24,735 [63] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2025-02-07 05:48:24,735 [63] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 06:48:24.735445+00:00 (in 3599.999548 seconds)
globalpromstats stdout | 2025-02-07 05:48:24,736 [63] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-02-07 06:48:24 UTC)" (scheduled at 2025-02-07 05:48:24.735445+00:00)
globalpromstats stdout | 2025-02-07 05:48:24,736 [63] [DEBUG] [__main__] Attempting to report stats
globalpromstats stdout | 2025-02-07 05:48:24,736 [63] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 05:48:24,736 [63] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ...
globalpromstats stdout | 2025-02-07 05:48:24,739 [63] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2025-02-07 05:48:24,739 [63] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 05:48:24,739 [63] [DEBUG] [__main__] Reporting global stats
globalpromstats stdout | 2025-02-07 05:48:24,739 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2025-02-07 05:48:24,746 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False])
globalpromstats stdout | 2025-02-07 05:48:24,748 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True])
globalpromstats stdout | 2025-02-07 05:48:24,749 [63] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2025-02-07 05:48:24,750 [63] [DEBUG] [data.database] Disconnecting from database.
globalpromstats stdout | 2025-02-07 05:48:24,750 [63] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 05:48:24,750 [63] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2025-02-07 05:48:24,751 [63] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS
globalpromstats stdout | 2025-02-07 05:48:24,751 [63] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2025-02-07 06:48:24 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:48:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:48:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:38.732147+00:00 (in 9.999498 seconds)
notificationworker stdout | 2025-02-07 05:48:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:38 UTC)" (scheduled at 2025-02-07 05:48:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:48:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:48:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 28, 732961), True, datetime.datetime(2025, 2, 7, 5, 48, 28, 732961), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:48:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:48:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:48:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:48:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:48:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:44.225378+00:00 (in 14.999570 seconds)
quotatotalworker stdout | 2025-02-07 05:48:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:44 UTC)" (scheduled at 2025-02-07 05:48:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:48:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:48:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:48:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:48:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:48:29,236 [237] [DEBUG] [app] Starting request: urn:request:ac63d543-ec6e-4ad1-9553-85bd7b33fadb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:29,236 [235] [DEBUG] [app] Starting request: urn:request:79ffb38a-bc0a-4070-b091-7b509e0c644c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:48:29,239 [246] [DEBUG] [app] Starting request: urn:request:a36ba6d0-df88-4e25-a239-35c35013a2b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:29,239 [245] [DEBUG] [app] Starting request: urn:request:06eff5c2-c2fc-43e8-9b51-9e1108f60b88 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:29,239 [246] [DEBUG] [app] Ending request: urn:request:a36ba6d0-df88-4e25-a239-35c35013a2b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a36ba6d0-df88-4e25-a239-35c35013a2b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:48:29,239 [245] [DEBUG] [app] Ending request: urn:request:06eff5c2-c2fc-43e8-9b51-9e1108f60b88 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:06eff5c2-c2fc-43e8-9b51-9e1108f60b88', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:48:29,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:48:29,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:48:29,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:29,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:29,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:29,242 [237] [DEBUG] [app] Starting request: urn:request:a694d72e-b771-446a-9357-181328196772 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:29,242 [238] [DEBUG] [app] Starting request: urn:request:8920dc98-afda-4098-8051-70463c0b1abf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:29,243 [237] [DEBUG] [app] Ending request: urn:request:a694d72e-b771-446a-9357-181328196772 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a694d72e-b771-446a-9357-181328196772', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:48:29,243 [238] [DEBUG] [app] Ending request: urn:request:8920dc98-afda-4098-8051-70463c0b1abf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8920dc98-afda-4098-8051-70463c0b1abf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:48:29,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:48:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:48:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:29,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:29,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:29,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:29,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:29,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:29,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:29,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:29,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:29,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:29,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:29,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:29,255 [235] [DEBUG] [app] Ending request: urn:request:79ffb38a-bc0a-4070-b091-7b509e0c644c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79ffb38a-bc0a-4070-b091-7b509e0c644c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:48:29,255 [237] [DEBUG] [app] Ending request: urn:request:ac63d543-ec6e-4ad1-9553-85bd7b33fadb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac63d543-ec6e-4ad1-9553-85bd7b33fadb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:48:29,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:48:29,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:48:29,256 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:48:29,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
namespacegcworker stdout | 2025-02-07 05:48:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:48:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:32.625945+00:00 (in 1.004352 seconds)
namespacegcworker stdout | 2025-02-07 05:48:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:31 UTC)" (scheduled at 2025-02-07 05:48:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:48:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:48:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 31, 621855), True, datetime.datetime(2025, 2, 7, 5, 48, 31, 621855), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:48:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:48:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:48:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:48:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:48:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:03.419110+00:00 (in 30.997968 seconds)
repositorygcworker stdout | 2025-02-07 05:48:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:53:32 UTC)" (scheduled at 2025-02-07 05:48:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:48:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 48, 32, 421406), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:48:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 48, 32, 421406), True, datetime.datetime(2025, 2, 7, 5, 48, 32, 421406), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:48:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 48, 32, 421406), True, datetime.datetime(2025, 2, 7, 5, 48, 32, 421406), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 48, 32, 421406), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:48:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:48:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:53:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:48:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:48:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:31.621131+00:00 (in 58.994738 seconds)
namespacegcworker stdout | 2025-02-07 05:48:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:32 UTC)" (scheduled at 2025-02-07 05:48:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:48:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:48:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:48:33,105 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:48:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:48:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:03.919286+00:00 (in 29.999534 seconds)
buildlogsarchiver stdout | 2025-02-07 05:48:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:03 UTC)" (scheduled at 2025-02-07 05:48:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:48:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 48, 33, 920000), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:48:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:48:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:48:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:48:34,727 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:48:35,878 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:56.713547+00:00 (in 18.997188 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:37 UTC)" (scheduled at 2025-02-07 05:48:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:48:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:48:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:48.732147+00:00 (in 9.999521 seconds)
notificationworker stdout | 2025-02-07 05:48:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:48 UTC)" (scheduled at 2025-02-07 05:48:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:48:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:48:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 38, 732824), True, datetime.datetime(2025, 2, 7, 5, 48, 38, 732824), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:48:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:48:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:48:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:48:40,142 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:48:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:48:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:54.423055+00:00 (in 13.998366 seconds)
gcworker stdout | 2025-02-07 05:48:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:10 UTC)" (scheduled at 2025-02-07 05:48:40.424250+00:00)
gcworker stdout | 2025-02-07 05:48:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:48:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907020430, None, 1, 0])
gcworker stdout | 2025-02-07 05:48:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:48:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:48:41,218 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:48:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:48:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:11.831521+00:00 (in 29.999560 seconds)
autopruneworker stdout | 2025-02-07 05:48:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:11 UTC)" (scheduled at 2025-02-07 05:48:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:48:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903721836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:48:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:48:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:48:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:48:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:48:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:59.225378+00:00 (in 14.999557 seconds)
quotatotalworker stdout | 2025-02-07 05:48:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:59 UTC)" (scheduled at 2025-02-07 05:48:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:48:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:48:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:48:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:48:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:48:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:48:44,235 [238] [DEBUG] [app] Starting request: urn:request:30a63c12-0fb6-4ea5-bf50-6dee727c7fe6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:44,235 [235] [DEBUG] [app] Starting request: urn:request:6b7f8663-991b-4599-8c50-6ba493adcc46 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:48:44,239 [246] [DEBUG] [app] Starting request: urn:request:5e412fee-fd2e-4334-9636-c159603ac6ca (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:44,239 [239] [DEBUG] [app] Starting request: urn:request:aa05fd44-33cd-4881-83dc-37fb547211ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:44,239 [239] [DEBUG] [app] Ending request: urn:request:aa05fd44-33cd-4881-83dc-37fb547211ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa05fd44-33cd-4881-83dc-37fb547211ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:48:44,239 [246] [DEBUG] [app] Ending request: urn:request:5e412fee-fd2e-4334-9636-c159603ac6ca (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5e412fee-fd2e-4334-9636-c159603ac6ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:48:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:48:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:48:44,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:44,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:44,242 [238] [DEBUG] [app] Starting request: urn:request:a6cbf9cf-9991-49e5-8275-fe12c219c878 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:44,242 [238] [DEBUG] [app] Ending request: urn:request:a6cbf9cf-9991-49e5-8275-fe12c219c878 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a6cbf9cf-9991-49e5-8275-fe12c219c878', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:48:44,243 [237] [DEBUG] [app] Starting request: urn:request:896afc29-3bff-4b18-9cbd-2a16e20460e3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:48:44,243 [237] [DEBUG] [app] Ending request: urn:request:896afc29-3bff-4b18-9cbd-2a16e20460e3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:896afc29-3bff-4b18-9cbd-2a16e20460e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:48:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:48:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:44,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:44,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:44,255 [238] [DEBUG] [app] Ending request: urn:request:30a63c12-0fb6-4ea5-bf50-6dee727c7fe6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:30a63c12-0fb6-4ea5-bf50-6dee727c7fe6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:48:44,255 [235] [DEBUG] [app] Ending request: urn:request:6b7f8663-991b-4599-8c50-6ba493adcc46 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b7f8663-991b-4599-8c50-6ba493adcc46', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:48:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:48:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:48:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:48:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:48:45,274 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:48:45,767 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:48:46,330 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:48:47,731 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:48:48,071 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:48:48,074 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:48:48,077 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:48:48,079 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:48:48,083 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:48:48,475 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:48:48,537 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:48:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:48:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:50.814677+00:00 (in 2.082075 seconds)
notificationworker stdout | 2025-02-07 05:48:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:58 UTC)" (scheduled at 2025-02-07 05:48:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:48:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:48:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 48, 732816), True, datetime.datetime(2025, 2, 7, 5, 48, 48, 732816), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:48:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:48:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:48:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:48:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:48:49,086 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:48:49,089 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:48:49,092 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:48:49,879 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:50,062 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:48:50,505 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:48:50,573 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:48:50,576 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:48:50,579 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:48:50,582 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:48:50,587 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:48:50,589 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:48:50,593 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:48:50,602 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:48:50,605 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:48:50,654 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
notificationworker stdout | 2025-02-07 05:48:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:48:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:48:58.732147+00:00 (in 7.917015 seconds)
notificationworker stdout | 2025-02-07 05:48:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:50 UTC)" (scheduled at 2025-02-07 05:48:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:48:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:48:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:48:50,846 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:48:51,249 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:48:51,629 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
securityscanningnotificationworker stdout | 2025-02-07 05:48:51,938 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
manifestbackfillworker stdout | 2025-02-07 05:48:51,961 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
buildlogsarchiver stdout | 2025-02-07 05:48:53,676 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:48:53,721 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:48:54,124 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:48:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:48:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:10.424250+00:00 (in 16.000740 seconds)
gcworker stdout | 2025-02-07 05:48:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:24 UTC)" (scheduled at 2025-02-07 05:48:54.423055+00:00)
gcworker stdout | 2025-02-07 05:48:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:48:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:48:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:48:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:56.118855+00:00 (in 59.999502 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:48:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:56 UTC)" (scheduled at 2025-02-07 05:48:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:48:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:48:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:48:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:37.715878+00:00 (in 41.001862 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:56 UTC)" (scheduled at 2025-02-07 05:48:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:48:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:48:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:48:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:08.732147+00:00 (in 9.999531 seconds)
notificationworker stdout | 2025-02-07 05:48:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:08 UTC)" (scheduled at 2025-02-07 05:48:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:48:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:48:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 48, 58, 732928), True, datetime.datetime(2025, 2, 7, 5, 48, 58, 732928), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:48:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:48:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:48:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:48:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:48:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:14.225378+00:00 (in 14.999550 seconds)
quotatotalworker stdout | 2025-02-07 05:48:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:14 UTC)" (scheduled at 2025-02-07 05:48:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:48:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:48:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:48:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:48:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:48:59,235 [238] [DEBUG] [app] Starting request: urn:request:0c422fb1-eeb3-4ce9-bc69-878a8be91ba6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:59,235 [237] [DEBUG] [app] Starting request: urn:request:c82443c7-c064-4327-b884-11d8b3875691 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:48:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:48:59,238 [239] [DEBUG] [app] Starting request: urn:request:f0041425-724d-40d4-a26a-cc947f4c325d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:59,239 [246] [DEBUG] [app] Starting request: urn:request:a328307e-5c19-49db-9f3a-acd9ddfc1611 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:48:59,239 [239] [DEBUG] [app] Ending request: urn:request:f0041425-724d-40d4-a26a-cc947f4c325d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0041425-724d-40d4-a26a-cc947f4c325d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:48:59,239 [246] [DEBUG] [app] Ending request: urn:request:a328307e-5c19-49db-9f3a-acd9ddfc1611 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a328307e-5c19-49db-9f3a-acd9ddfc1611', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:48:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:48:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:48:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:59,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:48:59,242 [238] [DEBUG] [app] Starting request: urn:request:ac495cad-c602-4015-a4ce-dc726b6770f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:59,242 [238] [DEBUG] [app] Ending request: urn:request:ac495cad-c602-4015-a4ce-dc726b6770f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ac495cad-c602-4015-a4ce-dc726b6770f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:48:59,243 [235] [DEBUG] [app] Starting request: urn:request:0d4d38ba-700b-4af9-bf9a-d4ae25c8f94a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:48:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:48:59,243 [235] [DEBUG] [app] Ending request: urn:request:0d4d38ba-700b-4af9-bf9a-d4ae25c8f94a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0d4d38ba-700b-4af9-bf9a-d4ae25c8f94a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:48:59,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:48:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:48:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:48:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:48:59,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:48:59,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:59,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:48:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:48:59,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:59,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:48:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:48:59,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:59,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:48:59,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:59,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:48:59,255 [238] [DEBUG] [app] Ending request: urn:request:0c422fb1-eeb3-4ce9-bc69-878a8be91ba6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0c422fb1-eeb3-4ce9-bc69-878a8be91ba6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:48:59,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:48:59,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:48:59,256 [237] [DEBUG] [app] Ending request: urn:request:c82443c7-c064-4327-b884-11d8b3875691 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c82443c7-c064-4327-b884-11d8b3875691', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 05:48:59,256 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:48:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:48:59,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:48:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
exportactionlogsworker stdout | 2025-02-07 05:48:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:48:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:04.936688+00:00 (in 4.998308 seconds)
exportactionlogsworker stdout | 2025-02-07 05:48:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:59 UTC)" (scheduled at 2025-02-07 05:48:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:48:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:48:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:49:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:49:03,139 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:49:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:49:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:23.420886+00:00 (in 20.001302 seconds)
repositorygcworker stdout | 2025-02-07 05:49:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:03 UTC)" (scheduled at 2025-02-07 05:49:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:49:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:49:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 3, 419860), True, datetime.datetime(2025, 2, 7, 5, 49, 3, 419860), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:49:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:49:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:49:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:49:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:49:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:33.919286+00:00 (in 29.999550 seconds)
buildlogsarchiver stdout | 2025-02-07 05:49:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:33 UTC)" (scheduled at 2025-02-07 05:49:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:49:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 49, 3, 919982), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:49:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:49:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:49:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:49:04,763 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:49:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:49:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:52.937648+00:00 (in 48.000499 seconds)
exportactionlogsworker stdout | 2025-02-07 05:49:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:04 UTC)" (scheduled at 2025-02-07 05:49:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:49:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:49:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 4, 937469), True, datetime.datetime(2025, 2, 7, 5, 49, 4, 937469), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:49:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:49:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:49:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:49:05,891 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:49:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:49:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:18.732147+00:00 (in 9.999504 seconds)
notificationworker stdout | 2025-02-07 05:49:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:18 UTC)" (scheduled at 2025-02-07 05:49:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:49:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:49:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 8, 732941), True, datetime.datetime(2025, 2, 7, 5, 49, 8, 732941), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:49:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:49:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:49:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:49:10,178 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:49:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:49:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:24.423055+00:00 (in 13.998392 seconds)
gcworker stdout | 2025-02-07 05:49:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:40 UTC)" (scheduled at 2025-02-07 05:49:10.424250+00:00)
gcworker stdout | 2025-02-07 05:49:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:49:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907050430, None, 1, 0])
gcworker stdout | 2025-02-07 05:49:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:49:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:49:11,249 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:49:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:49:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:41.831521+00:00 (in 29.999578 seconds)
autopruneworker stdout | 2025-02-07 05:49:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:41 UTC)" (scheduled at 2025-02-07 05:49:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:49:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903751836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:49:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:49:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:49:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:49:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:49:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:29.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 05:49:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:29 UTC)" (scheduled at 2025-02-07 05:49:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:49:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:49:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:49:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:49:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:49:14,236 [238] [DEBUG] [app] Starting request: urn:request:e876d71c-c02c-4df9-99f9-b2d518bd6e7d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:14,236 [237] [DEBUG] [app] Starting request: urn:request:cb80e94b-8738-4e1b-85cf-c6b077c68459 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:49:14,238 [246] [DEBUG] [app] Starting request: urn:request:6dd76e9f-37a5-4910-ad0d-5996a54d2ebb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:14,238 [239] [DEBUG] [app] Starting request: urn:request:a057121b-3870-4ae6-80b7-f0024704f57d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:14,239 [239] [DEBUG] [app] Ending request: urn:request:a057121b-3870-4ae6-80b7-f0024704f57d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a057121b-3870-4ae6-80b7-f0024704f57d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:49:14,239 [246] [DEBUG] [app] Ending request: urn:request:6dd76e9f-37a5-4910-ad0d-5996a54d2ebb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6dd76e9f-37a5-4910-ad0d-5996a54d2ebb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:49:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:49:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:14,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:14,242 [238] [DEBUG] [app] Starting request: urn:request:75095669-22c2-49d3-9023-2060f499e20b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:49:14,242 [238] [DEBUG] [app] Ending request: urn:request:75095669-22c2-49d3-9023-2060f499e20b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:75095669-22c2-49d3-9023-2060f499e20b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:49:14,242 [235] [DEBUG] [app] Starting request: urn:request:432a3063-ac4b-4504-aa0a-34da90f23341 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:49:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:14,243 [235] [DEBUG] [app] Ending request: urn:request:432a3063-ac4b-4504-aa0a-34da90f23341 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:432a3063-ac4b-4504-aa0a-34da90f23341', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:49:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:49:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:14,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:14,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:49:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:49:14,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:14,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:14,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:14,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:14,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:14,251 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:14,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:14,253 [238] [DEBUG] [app] Ending request: urn:request:e876d71c-c02c-4df9-99f9-b2d518bd6e7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e876d71c-c02c-4df9-99f9-b2d518bd6e7d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:14,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:49:14,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:49:14,254 [237] [DEBUG] [app] Ending request: urn:request:cb80e94b-8738-4e1b-85cf-c6b077c68459 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb80e94b-8738-4e1b-85cf-c6b077c68459', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:14,254 [237] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:49:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
quotatotalworker stdout | 2025-02-07 05:49:15,310 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:49:15,803 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:49:16,343 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:49:17,767 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:49:18,079 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:49:18,082 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:49:18,085 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:49:18,089 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:49:18,092 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:49:18,511 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:49:18,573 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:49:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:49:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:28.732147+00:00 (in 9.999564 seconds)
notificationworker stdout | 2025-02-07 05:49:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:28 UTC)" (scheduled at 2025-02-07 05:49:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:49:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:49:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 18, 732845), True, datetime.datetime(2025, 2, 7, 5, 49, 18, 732845), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:49:18,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:49:18,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:49:18,743 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:49:19,093 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:49:19,097 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:49:19,100 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:49:19,888 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:20,093 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:49:20,541 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:49:20,581 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:49:20,586 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:49:20,590 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:49:20,593 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:49:20,597 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:49:20,600 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:49:20,603 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:49:20,610 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:49:20,613 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:49:20,662 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:49:20,877 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:49:21,275 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:49:21,644 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:49:21,969 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:49:21,973 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
repositorygcworker stdout | 2025-02-07 05:49:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:49:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:03.419110+00:00 (in 39.997768 seconds)
repositorygcworker stdout | 2025-02-07 05:49:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:23 UTC)" (scheduled at 2025-02-07 05:49:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:49:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:49:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:49:23,709 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:49:23,757 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:49:24,155 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:49:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:49:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:40.424250+00:00 (in 16.000751 seconds)
gcworker stdout | 2025-02-07 05:49:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:54 UTC)" (scheduled at 2025-02-07 05:49:24.423055+00:00)
gcworker stdout | 2025-02-07 05:49:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:49:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:49:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:49:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:49:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:38.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 05:49:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:38 UTC)" (scheduled at 2025-02-07 05:49:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:49:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:49:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 28, 732806), True, datetime.datetime(2025, 2, 7, 5, 49, 28, 732806), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:49:28,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:49:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:49:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:49:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:49:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:44.225378+00:00 (in 14.999530 seconds)
quotatotalworker stdout | 2025-02-07 05:49:29,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:44 UTC)" (scheduled at 2025-02-07 05:49:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:49:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:49:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:49:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:49:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:49:29,235 [238] [DEBUG] [app] Starting request: urn:request:e8f5bf81-18e0-4035-97a4-1114570f7d23 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:29,235 [237] [DEBUG] [app] Starting request: urn:request:7a856c5c-33a4-4da3-8a4d-83d5b5433ddd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:29,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:49:29,238 [246] [DEBUG] [app] Starting request: urn:request:5844d8d2-9e56-4dc7-befc-6cd5c87e1070 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:29,238 [245] [DEBUG] [app] Starting request: urn:request:e16067f5-b386-4212-8976-39e882ab8345 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:29,238 [246] [DEBUG] [app] Ending request: urn:request:5844d8d2-9e56-4dc7-befc-6cd5c87e1070 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5844d8d2-9e56-4dc7-befc-6cd5c87e1070', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:49:29,238 [245] [DEBUG] [app] Ending request: urn:request:e16067f5-b386-4212-8976-39e882ab8345 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e16067f5-b386-4212-8976-39e882ab8345', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:49:29,238 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:49:29,238 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:29,238 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:29,238 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:29,242 [236] [DEBUG] [app] Starting request: urn:request:69d478f5-21f2-44b4-90e2-892d3638ceff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:49:29,242 [236] [DEBUG] [app] Ending request: urn:request:69d478f5-21f2-44b4-90e2-892d3638ceff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:69d478f5-21f2-44b4-90e2-892d3638ceff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:49:29,242 [235] [DEBUG] [app] Starting request: urn:request:4f8a801e-07f8-47d4-92bd-43c5654bba3a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:49:29,243 [235] [DEBUG] [app] Ending request: urn:request:4f8a801e-07f8-47d4-92bd-43c5654bba3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f8a801e-07f8-47d4-92bd-43c5654bba3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:49:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:29,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:29,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:49:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:29,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:49:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:29,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:49:29,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:29,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:29,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:29,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:29,254 [238] [DEBUG] [app] Ending request: urn:request:e8f5bf81-18e0-4035-97a4-1114570f7d23 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e8f5bf81-18e0-4035-97a4-1114570f7d23', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:49:29,254 [237] [DEBUG] [app] Ending request: urn:request:7a856c5c-33a4-4da3-8a4d-83d5b5433ddd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7a856c5c-33a4-4da3-8a4d-83d5b5433ddd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:29,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:49:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.021)
gunicorn-web stdout | 2025-02-07 05:49:29,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:49:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:49:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:32.625945+00:00 (in 1.004354 seconds)
namespacegcworker stdout | 2025-02-07 05:49:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:31 UTC)" (scheduled at 2025-02-07 05:49:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:49:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:49:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 31, 621896), True, datetime.datetime(2025, 2, 7, 5, 49, 31, 621896), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:49:31,630 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:49:31,630 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:49:31,630 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:49:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:49:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:31.621131+00:00 (in 58.994710 seconds)
namespacegcworker stdout | 2025-02-07 05:49:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:32 UTC)" (scheduled at 2025-02-07 05:49:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:49:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:49:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:49:33,162 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:49:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:49:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:03.919286+00:00 (in 29.999565 seconds)
buildlogsarchiver stdout | 2025-02-07 05:49:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:03 UTC)" (scheduled at 2025-02-07 05:49:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:49:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 49, 33, 919965), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:49:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:49:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:49:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:49:34,772 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:49:35,927 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:56.713547+00:00 (in 18.997173 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:37 UTC)" (scheduled at 2025-02-07 05:49:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:49:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:49:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:48.732147+00:00 (in 9.999555 seconds)
notificationworker stdout | 2025-02-07 05:49:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:48 UTC)" (scheduled at 2025-02-07 05:49:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:49:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:49:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 38, 732795), True, datetime.datetime(2025, 2, 7, 5, 49, 38, 732795), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:49:38,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:49:38,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:49:38,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:49:40,215 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:49:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:49:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:54.423055+00:00 (in 13.998388 seconds)
gcworker stdout | 2025-02-07 05:49:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:10 UTC)" (scheduled at 2025-02-07 05:49:40.424250+00:00)
gcworker stdout | 2025-02-07 05:49:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:49:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907080431, None, 1, 0])
gcworker stdout | 2025-02-07 05:49:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:49:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:49:41,274 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:49:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:49:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:11.831521+00:00 (in 29.999573 seconds)
autopruneworker stdout | 2025-02-07 05:49:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:11 UTC)" (scheduled at 2025-02-07 05:49:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:49:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903781836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:49:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:49:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:49:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:49:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:49:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:59.225378+00:00 (in 14.999507 seconds)
quotatotalworker stdout | 2025-02-07 05:49:44,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:59 UTC)" (scheduled at 2025-02-07 05:49:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:49:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:49:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:49:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:49:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:49:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:49:44,235 [236] [DEBUG] [app] Starting request: urn:request:27a635f6-644b-4940-ad74-01853a984460 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:44,235 [238] [DEBUG] [app] Starting request: urn:request:9aae0a6e-80a4-443c-b451-b0b757b3a09b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:49:44,238 [239] [DEBUG] [app] Starting request: urn:request:6f49d213-2809-4ef2-be62-a42e21af71c8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:44,238 [245] [DEBUG] [app] Starting request: urn:request:8d82cb7d-c827-47d7-87bd-4a425c67c8c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:44,239 [239] [DEBUG] [app] Ending request: urn:request:6f49d213-2809-4ef2-be62-a42e21af71c8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6f49d213-2809-4ef2-be62-a42e21af71c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:49:44,239 [245] [DEBUG] [app] Ending request: urn:request:8d82cb7d-c827-47d7-87bd-4a425c67c8c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d82cb7d-c827-47d7-87bd-4a425c67c8c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:49:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:49:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:44,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:44,242 [237] [DEBUG] [app] Starting request: urn:request:f922f936-aaaf-459e-8812-f547968dd34a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:49:44,242 [237] [DEBUG] [app] Ending request: urn:request:f922f936-aaaf-459e-8812-f547968dd34a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f922f936-aaaf-459e-8812-f547968dd34a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:49:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:44,243 [235] [DEBUG] [app] Starting request: urn:request:01d3444f-8b49-426a-9903-87b3563a570e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:49:44,243 [235] [DEBUG] [app] Ending request: urn:request:01d3444f-8b49-426a-9903-87b3563a570e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:01d3444f-8b49-426a-9903-87b3563a570e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:49:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:49:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:44,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:49:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:44,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:44,254 [238] [DEBUG] [app] Ending request: urn:request:9aae0a6e-80a4-443c-b451-b0b757b3a09b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9aae0a6e-80a4-443c-b451-b0b757b3a09b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
gunicorn-web stdout | 2025-02-07 05:49:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:49:44,255 [236] [DEBUG] [app] Ending request: urn:request:27a635f6-644b-4940-ad74-01853a984460 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:27a635f6-644b-4940-ad74-01853a984460', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:49:44,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.022 118 0.022)
quotatotalworker stdout | 2025-02-07 05:49:45,331 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:49:45,840 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:49:46,379 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:49:47,773 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:49:48,088 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:49:48,091 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:49:48,093 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:49:48,096 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:49:48,099 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:49:48,547 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:49:48,609 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:49:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:49:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:50.814677+00:00 (in 2.082081 seconds)
notificationworker stdout | 2025-02-07 05:49:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:58 UTC)" (scheduled at 2025-02-07 05:49:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:49:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:49:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 48, 732797), True, datetime.datetime(2025, 2, 7, 5, 49, 48, 732797), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:49:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:49:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:49:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:49:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:49:49,101 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:49:49,105 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:49:49,108 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:49:49,917 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:50,129 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:49:50,547 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:49:50,589 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:49:50,598 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:49:50,601 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:49:50,603 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:49:50,606 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:49:50,610 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:49:50,612 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:49:50,617 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:49:50,621 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:49:50,698 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
notificationworker stdout | 2025-02-07 05:49:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:49:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:58.732147+00:00 (in 7.916997 seconds)
notificationworker stdout | 2025-02-07 05:49:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:50 UTC)" (scheduled at 2025-02-07 05:49:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:49:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:49:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:49:50,913 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:49:51,290 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:49:51,680 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:49:52,006 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:49:52,010 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
exportactionlogsworker stdout | 2025-02-07 05:49:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:49:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:49:59.937899+00:00 (in 6.999761 seconds)
exportactionlogsworker stdout | 2025-02-07 05:49:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:54:52 UTC)" (scheduled at 2025-02-07 05:49:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:49:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 49, 52, 938385), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:49:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 49, 52, 938385), True, datetime.datetime(2025, 2, 7, 5, 49, 52, 938385), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:49:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 49, 52, 938385), True, datetime.datetime(2025, 2, 7, 5, 49, 52, 938385), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 49, 52, 938385), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:49:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:49:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:54:52 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:49:53,734 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:49:53,774 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:49:54,179 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:49:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:49:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:10.424250+00:00 (in 16.000752 seconds)
gcworker stdout | 2025-02-07 05:49:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:24 UTC)" (scheduled at 2025-02-07 05:49:54.423055+00:00)
gcworker stdout | 2025-02-07 05:49:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:49:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:49:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:49:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:56.118855+00:00 (in 59.999492 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:49:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:56 UTC)" (scheduled at 2025-02-07 05:49:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:49:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:49:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:49:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:37.715878+00:00 (in 41.001853 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:56 UTC)" (scheduled at 2025-02-07 05:49:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:49:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:49:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:49:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:08.732147+00:00 (in 9.999518 seconds)
notificationworker stdout | 2025-02-07 05:49:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:08 UTC)" (scheduled at 2025-02-07 05:49:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:49:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:49:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 49, 58, 732896), True, datetime.datetime(2025, 2, 7, 5, 49, 58, 732896), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:49:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:49:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:49:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:49:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:49:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:14.225378+00:00 (in 14.999579 seconds)
quotatotalworker stdout | 2025-02-07 05:49:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:14 UTC)" (scheduled at 2025-02-07 05:49:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:49:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:49:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:49:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:49:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:49:59,236 [235] [DEBUG] [app] Starting request: urn:request:c30def5a-9866-4eb6-bd52-a3bd7528d98f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:59,236 [236] [DEBUG] [app] Starting request: urn:request:1462eb9a-27b2-40af-86ab-8324b4c65ca7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:49:59,238 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:59,238 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:49:59,240 [239] [DEBUG] [app] Starting request: urn:request:d3b6d1fb-5590-4567-be29-e114fb444300 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:59,240 [246] [DEBUG] [app] Starting request: urn:request:ca5f05a1-3663-4eed-8710-a20b25af6f0c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:49:59,240 [239] [DEBUG] [app] Ending request: urn:request:d3b6d1fb-5590-4567-be29-e114fb444300 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d3b6d1fb-5590-4567-be29-e114fb444300', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:49:59,240 [246] [DEBUG] [app] Ending request: urn:request:ca5f05a1-3663-4eed-8710-a20b25af6f0c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ca5f05a1-3663-4eed-8710-a20b25af6f0c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:49:59,240 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:49:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:49:59,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:49:59,241 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:59,242 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:59,242 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:49:59,243 [235] [DEBUG] [app] Starting request: urn:request:f2137ba7-828b-4168-9ee6-e2c47f72b848 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:49:59,243 [238] [DEBUG] [app] Starting request: urn:request:d322bf5b-8955-4eb6-bc01-b593b05bbca4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:49:59,243 [235] [DEBUG] [app] Ending request: urn:request:f2137ba7-828b-4168-9ee6-e2c47f72b848 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2137ba7-828b-4168-9ee6-e2c47f72b848', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:49:59,243 [238] [DEBUG] [app] Ending request: urn:request:d322bf5b-8955-4eb6-bc01-b593b05bbca4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d322bf5b-8955-4eb6-bc01-b593b05bbca4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:49:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:49:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:59,244 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:49:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:49:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:49:59,244 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:49:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:49:59,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:49:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:49:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:59,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:49:59,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:49:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:59,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:49:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:49:59,254 [235] [DEBUG] [app] Ending request: urn:request:c30def5a-9866-4eb6-bd52-a3bd7528d98f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c30def5a-9866-4eb6-bd52-a3bd7528d98f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:59,254 [236] [DEBUG] [app] Ending request: urn:request:1462eb9a-27b2-40af-86ab-8324b4c65ca7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1462eb9a-27b2-40af-86ab-8324b4c65ca7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:49:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:49:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:49:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:49:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:49:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:49:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:49:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:49:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:04.936688+00:00 (in 4.998343 seconds)
exportactionlogsworker stdout | 2025-02-07 05:49:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:59 UTC)" (scheduled at 2025-02-07 05:49:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:49:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:49:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:50:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:50:03,198 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:50:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:50:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:23.420886+00:00 (in 20.001345 seconds)
repositorygcworker stdout | 2025-02-07 05:50:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:03 UTC)" (scheduled at 2025-02-07 05:50:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:50:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:50:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 3, 419812), True, datetime.datetime(2025, 2, 7, 5, 50, 3, 419812), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:50:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:50:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:50:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:50:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:50:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:33.919286+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2025-02-07 05:50:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:33 UTC)" (scheduled at 2025-02-07 05:50:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:50:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 50, 3, 919982), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:50:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:50:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:50:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:50:04,803 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:50:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:50:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:59.937899+00:00 (in 55.000786 seconds)
exportactionlogsworker stdout | 2025-02-07 05:50:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:04 UTC)" (scheduled at 2025-02-07 05:50:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:50:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:50:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 4, 937416), True, datetime.datetime(2025, 2, 7, 5, 50, 4, 937416), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:50:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:50:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:50:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:50:05,952 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:50:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:50:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:18.732147+00:00 (in 9.999561 seconds)
notificationworker stdout | 2025-02-07 05:50:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:18 UTC)" (scheduled at 2025-02-07 05:50:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:50:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:50:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 8, 732877), True, datetime.datetime(2025, 2, 7, 5, 50, 8, 732877), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:50:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:50:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:50:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:50:10,231 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:50:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:50:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:24.423055+00:00 (in 13.998404 seconds)
gcworker stdout | 2025-02-07 05:50:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:40 UTC)" (scheduled at 2025-02-07 05:50:10.424250+00:00)
gcworker stdout | 2025-02-07 05:50:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:50:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907110431, None, 1, 0])
gcworker stdout | 2025-02-07 05:50:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:50:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:50:11,283 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:50:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:50:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:41.831521+00:00 (in 29.999533 seconds)
autopruneworker stdout | 2025-02-07 05:50:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:41 UTC)" (scheduled at 2025-02-07 05:50:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:50:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903811836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:50:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:50:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:50:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:50:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:50:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:29.225378+00:00 (in 14.999559 seconds)
quotatotalworker stdout | 2025-02-07 05:50:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:29 UTC)" (scheduled at 2025-02-07 05:50:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:50:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:50:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:50:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:50:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:50:14,235 [236] [DEBUG] [app] Starting request: urn:request:64d47fbc-4b10-4592-9797-27a2403ad6af (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:14,235 [238] [DEBUG] [app] Starting request: urn:request:9431b414-b723-48f5-a420-dc42f47d3099 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:50:14,238 [246] [DEBUG] [app] Starting request: urn:request:ab073a83-597f-4c03-9098-46f6f6b6200e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:14,239 [245] [DEBUG] [app] Starting request: urn:request:67993802-5753-426a-b87c-4cbed0c69342 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:14,239 [246] [DEBUG] [app] Ending request: urn:request:ab073a83-597f-4c03-9098-46f6f6b6200e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab073a83-597f-4c03-9098-46f6f6b6200e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:50:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:50:14,239 [245] [DEBUG] [app] Ending request: urn:request:67993802-5753-426a-b87c-4cbed0c69342 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:67993802-5753-426a-b87c-4cbed0c69342', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:50:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-web stdout | 2025-02-07 05:50:14,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:14,242 [236] [DEBUG] [app] Starting request: urn:request:57c0ad8b-00de-4549-98ab-a1aa1099357e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:50:14,242 [236] [DEBUG] [app] Ending request: urn:request:57c0ad8b-00de-4549-98ab-a1aa1099357e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57c0ad8b-00de-4549-98ab-a1aa1099357e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:50:14,242 [235] [DEBUG] [app] Starting request: urn:request:2b563642-0067-4b55-8e93-df28d2b5f592 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.002)
gunicorn-web stdout | 2025-02-07 05:50:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:14,243 [235] [DEBUG] [app] Ending request: urn:request:2b563642-0067-4b55-8e93-df28d2b5f592 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2b563642-0067-4b55-8e93-df28d2b5f592', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:50:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:14,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:14,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:14,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:14,255 [236] [DEBUG] [app] Ending request: urn:request:64d47fbc-4b10-4592-9797-27a2403ad6af (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:64d47fbc-4b10-4592-9797-27a2403ad6af', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:14,255 [238] [DEBUG] [app] Ending request: urn:request:9431b414-b723-48f5-a420-dc42f47d3099 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9431b414-b723-48f5-a420-dc42f47d3099', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:50:14,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:50:14,256 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:50:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:50:15,354 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:50:15,875 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:50:16,394 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:50:17,810 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:50:18,098 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:50:18,101 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:50:18,104 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:50:18,107 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:50:18,110 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:50:18,579 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:50:18,617 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:50:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:50:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:28.732147+00:00 (in 9.999550 seconds)
notificationworker stdout | 2025-02-07 05:50:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:28 UTC)" (scheduled at 2025-02-07 05:50:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:50:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:50:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 18, 732875), True, datetime.datetime(2025, 2, 7, 5, 50, 18, 732875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:50:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:50:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:50:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:50:19,109 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:50:19,111 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:50:19,115 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:50:19,954 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:20,165 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:50:20,556 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:50:20,601 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:50:20,609 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:50:20,613 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:50:20,615 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:50:20,618 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:50:20,622 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:50:20,625 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:50:20,629 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:50:20,635 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:50:20,726 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:50:20,931 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:50:21,326 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:50:21,706 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:50:22,015 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:50:22,045 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
repositorygcworker stdout | 2025-02-07 05:50:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:50:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:03.419110+00:00 (in 39.997789 seconds)
repositorygcworker stdout | 2025-02-07 05:50:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:23 UTC)" (scheduled at 2025-02-07 05:50:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:50:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:50:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:50:23,759 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:50:23,810 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:50:24,206 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:50:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:50:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:40.424250+00:00 (in 16.000766 seconds)
gcworker stdout | 2025-02-07 05:50:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:54 UTC)" (scheduled at 2025-02-07 05:50:24.423055+00:00)
gcworker stdout | 2025-02-07 05:50:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:50:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:50:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:50:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:50:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:38.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 05:50:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:38 UTC)" (scheduled at 2025-02-07 05:50:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:50:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:50:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 28, 732926), True, datetime.datetime(2025, 2, 7, 5, 50, 28, 732926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:50:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:50:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:50:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:50:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:50:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:44.225378+00:00 (in 14.999571 seconds)
quotatotalworker stdout | 2025-02-07 05:50:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:44 UTC)" (scheduled at 2025-02-07 05:50:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:50:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:50:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:50:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:50:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:50:29,235 [235] [DEBUG] [app] Starting request: urn:request:c73ebaae-2439-4afe-85fc-d02b6e5d40c7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:29,235 [238] [DEBUG] [app] Starting request: urn:request:4b47947a-ced5-4580-9877-20e8258eda0a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:50:29,238 [239] [DEBUG] [app] Starting request: urn:request:6eac3478-e85d-4636-9d88-281f0da12ec5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:29,238 [246] [DEBUG] [app] Starting request: urn:request:a138bc5d-cc68-4063-937e-5a8118e6b7d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:29,238 [239] [DEBUG] [app] Ending request: urn:request:6eac3478-e85d-4636-9d88-281f0da12ec5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6eac3478-e85d-4636-9d88-281f0da12ec5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:50:29,239 [246] [DEBUG] [app] Ending request: urn:request:a138bc5d-cc68-4063-937e-5a8118e6b7d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a138bc5d-cc68-4063-937e-5a8118e6b7d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:50:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:50:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:50:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:29,241 [238] [DEBUG] [app] Starting request: urn:request:a3d9521e-c2da-4a6f-90c5-eb2e1aaa1ed5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:50:29,241 [238] [DEBUG] [app] Ending request: urn:request:a3d9521e-c2da-4a6f-90c5-eb2e1aaa1ed5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a3d9521e-c2da-4a6f-90c5-eb2e1aaa1ed5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:50:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:29,242 [236] [DEBUG] [app] Starting request: urn:request:fba10ee0-7e80-409b-899e-f6aec4734c87 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:50:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:29,242 [236] [DEBUG] [app] Ending request: urn:request:fba10ee0-7e80-409b-899e-f6aec4734c87 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fba10ee0-7e80-409b-899e-f6aec4734c87', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.001)
gunicorn-web stdout | 2025-02-07 05:50:29,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:29,253 [238] [DEBUG] [app] Ending request: urn:request:4b47947a-ced5-4580-9877-20e8258eda0a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4b47947a-ced5-4580-9877-20e8258eda0a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
gunicorn-web stdout | 2025-02-07 05:50:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:50:29,254 [235] [DEBUG] [app] Ending request: urn:request:c73ebaae-2439-4afe-85fc-d02b6e5d40c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c73ebaae-2439-4afe-85fc-d02b6e5d40c7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 05:50:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:50:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:50:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:32.625945+00:00 (in 1.004366 seconds)
namespacegcworker stdout | 2025-02-07 05:50:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:31 UTC)" (scheduled at 2025-02-07 05:50:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:50:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:50:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 31, 621839), True, datetime.datetime(2025, 2, 7, 5, 50, 31, 621839), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:50:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:50:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:50:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:50:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:50:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:31.621131+00:00 (in 58.994704 seconds)
namespacegcworker stdout | 2025-02-07 05:50:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:32 UTC)" (scheduled at 2025-02-07 05:50:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:50:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:50:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:50:33,226 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:50:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:50:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:03.919286+00:00 (in 29.999532 seconds)
buildlogsarchiver stdout | 2025-02-07 05:50:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:03 UTC)" (scheduled at 2025-02-07 05:50:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:50:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 50, 33, 920007), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:50:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:50:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:50:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:50:34,839 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:50:35,986 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:56.713547+00:00 (in 18.997209 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:37 UTC)" (scheduled at 2025-02-07 05:50:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:37,722 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:50:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:50:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:48.732147+00:00 (in 9.999486 seconds)
notificationworker stdout | 2025-02-07 05:50:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:48 UTC)" (scheduled at 2025-02-07 05:50:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:50:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:50:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 38, 732978), True, datetime.datetime(2025, 2, 7, 5, 50, 38, 732978), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:50:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:50:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:50:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:50:40,267 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:50:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:50:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:54.423055+00:00 (in 13.998398 seconds)
gcworker stdout | 2025-02-07 05:50:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:10 UTC)" (scheduled at 2025-02-07 05:50:40.424250+00:00)
gcworker stdout | 2025-02-07 05:50:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:50:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907140430, None, 1, 0])
gcworker stdout | 2025-02-07 05:50:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:50:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:50:41,309 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:50:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:50:41,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:11.831521+00:00 (in 29.999500 seconds)
autopruneworker stdout | 2025-02-07 05:50:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:11 UTC)" (scheduled at 2025-02-07 05:50:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:50:41,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903841836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:50:41,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:50:41,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:50:41,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:50:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:50:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:59.225378+00:00 (in 14.999539 seconds)
quotatotalworker stdout | 2025-02-07 05:50:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:59 UTC)" (scheduled at 2025-02-07 05:50:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:50:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:50:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:50:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:50:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:50:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:50:44,236 [235] [DEBUG] [app] Starting request: urn:request:2a01f9f9-c1aa-47ba-8814-eadf9055c1a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:44,236 [236] [DEBUG] [app] Starting request: urn:request:109c979d-919c-43b9-9fcf-9b234e2249aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:44,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:50:44,239 [239] [DEBUG] [app] Starting request: urn:request:720f4f1a-88e4-4219-a102-d88ea2a92a70 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:44,239 [245] [DEBUG] [app] Starting request: urn:request:b6967c2d-43b2-4fd8-b47a-b60710451f3d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:44,239 [239] [DEBUG] [app] Ending request: urn:request:720f4f1a-88e4-4219-a102-d88ea2a92a70 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:720f4f1a-88e4-4219-a102-d88ea2a92a70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:50:44,239 [245] [DEBUG] [app] Ending request: urn:request:b6967c2d-43b2-4fd8-b47a-b60710451f3d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b6967c2d-43b2-4fd8-b47a-b60710451f3d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:50:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:50:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:50:44,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:44,240 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:44,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:44,242 [238] [DEBUG] [app] Starting request: urn:request:ac38b9b4-50b8-4272-866e-b5cce628c7ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:50:44,242 [238] [DEBUG] [app] Ending request: urn:request:ac38b9b4-50b8-4272-866e-b5cce628c7ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ac38b9b4-50b8-4272-866e-b5cce628c7ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:50:44,242 [235] [DEBUG] [app] Starting request: urn:request:95e8fc2c-5974-47f0-968b-ffb8803f13d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:50:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:44,243 [235] [DEBUG] [app] Ending request: urn:request:95e8fc2c-5974-47f0-968b-ffb8803f13d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95e8fc2c-5974-47f0-968b-ffb8803f13d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:50:44,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:44,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:44,244 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:44,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:44,248 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:44,248 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:44,252 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:44,254 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:44,254 [235] [DEBUG] [app] Ending request: urn:request:2a01f9f9-c1aa-47ba-8814-eadf9055c1a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a01f9f9-c1aa-47ba-8814-eadf9055c1a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:50:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:50:44,255 [236] [DEBUG] [app] Ending request: urn:request:109c979d-919c-43b9-9fcf-9b234e2249aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:109c979d-919c-43b9-9fcf-9b234e2249aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:44,255 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:50:44,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:50:45,370 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:50:45,912 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:50:46,419 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:50:47,832 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:50:48,108 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:50:48,112 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:50:48,115 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:50:48,119 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:50:48,122 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:50:48,588 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:50:48,645 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:50:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:50:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:50.814677+00:00 (in 2.082000 seconds)
notificationworker stdout | 2025-02-07 05:50:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:58 UTC)" (scheduled at 2025-02-07 05:50:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:50:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:50:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 48, 732978), True, datetime.datetime(2025, 2, 7, 5, 50, 48, 732978), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:50:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:50:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:50:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:50:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:50:49,116 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:50:49,119 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:50:49,123 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:50:49,965 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:50,197 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:50:50,579 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:50:50,612 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:50:50,617 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:50:50,621 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:50:50,624 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:50:50,627 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:50:50,630 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:50:50,634 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:50:50,637 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:50:50,641 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:50:50,762 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
notificationworker stdout | 2025-02-07 05:50:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:50:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:50:58.732147+00:00 (in 7.916959 seconds)
notificationworker stdout | 2025-02-07 05:50:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:50 UTC)" (scheduled at 2025-02-07 05:50:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:50:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:50:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:50 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:50:50,967 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:50:51,354 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:50:51,727 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:50:52,026 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:50:52,078 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
buildlogsarchiver stdout | 2025-02-07 05:50:53,788 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:50:53,827 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:50:54,242 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:50:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:50:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:10.424250+00:00 (in 16.000725 seconds)
gcworker stdout | 2025-02-07 05:50:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:24 UTC)" (scheduled at 2025-02-07 05:50:54.423055+00:00)
gcworker stdout | 2025-02-07 05:50:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:50:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:50:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:50:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:56.118855+00:00 (in 59.999525 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:50:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:56 UTC)" (scheduled at 2025-02-07 05:50:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:50:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:50:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:50:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:37.715878+00:00 (in 41.001874 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:56 UTC)" (scheduled at 2025-02-07 05:50:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:50:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:50:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:50:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:08.732147+00:00 (in 9.999505 seconds)
notificationworker stdout | 2025-02-07 05:50:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:08 UTC)" (scheduled at 2025-02-07 05:50:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:50:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:50:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 50, 58, 732948), True, datetime.datetime(2025, 2, 7, 5, 50, 58, 732948), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:50:58,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:50:58,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:50:58,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:50:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:50:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:14.225378+00:00 (in 14.999556 seconds)
quotatotalworker stdout | 2025-02-07 05:50:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:14 UTC)" (scheduled at 2025-02-07 05:50:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:50:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:50:59,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:50:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:50:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:50:59,235 [236] [DEBUG] [app] Starting request: urn:request:f18a9a59-a321-4da2-946a-4fce8c065ccb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:59,235 [235] [DEBUG] [app] Starting request: urn:request:12e7afdb-e662-41f0-8750-8137a34fd5e6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:50:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:59,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:50:59,238 [239] [DEBUG] [app] Starting request: urn:request:ee445017-6818-4732-8e78-209dccef29cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:59,238 [246] [DEBUG] [app] Starting request: urn:request:5095b70b-93bc-4783-9e63-f4a66da0dc19 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:50:59,239 [239] [DEBUG] [app] Ending request: urn:request:ee445017-6818-4732-8e78-209dccef29cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee445017-6818-4732-8e78-209dccef29cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:50:59,239 [246] [DEBUG] [app] Ending request: urn:request:5095b70b-93bc-4783-9e63-f4a66da0dc19 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5095b70b-93bc-4783-9e63-f4a66da0dc19', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:50:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:50:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2025-02-07 05:50:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:50:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:59,241 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:50:59,241 [238] [DEBUG] [app] Starting request: urn:request:f6379bf4-d58b-4715-b3e0-d01f36b74a5b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:50:59,242 [238] [DEBUG] [app] Ending request: urn:request:f6379bf4-d58b-4715-b3e0-d01f36b74a5b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f6379bf4-d58b-4715-b3e0-d01f36b74a5b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:50:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:50:59,242 [235] [DEBUG] [app] Starting request: urn:request:4a4f8393-1443-4428-b085-3cd3813a6ad4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:50:59,243 [235] [DEBUG] [app] Ending request: urn:request:4a4f8393-1443-4428-b085-3cd3813a6ad4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4a4f8393-1443-4428-b085-3cd3813a6ad4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:50:59,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:50:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:50:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:50:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:59,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:50:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:50:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:59,244 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:50:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:50:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:50:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:50:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:50:59,254 [235] [DEBUG] [app] Ending request: urn:request:12e7afdb-e662-41f0-8750-8137a34fd5e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:12e7afdb-e662-41f0-8750-8137a34fd5e6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:59,254 [236] [DEBUG] [app] Ending request: urn:request:f18a9a59-a321-4da2-946a-4fce8c065ccb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f18a9a59-a321-4da2-946a-4fce8c065ccb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:50:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:50:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:50:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:50:59,255 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:50:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:50:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:50:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:50:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:04.936688+00:00 (in 4.998333 seconds)
exportactionlogsworker stdout | 2025-02-07 05:50:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:59 UTC)" (scheduled at 2025-02-07 05:50:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:50:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:50:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:51:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:51:03,235 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:51:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:51:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:23.420886+00:00 (in 20.001336 seconds)
repositorygcworker stdout | 2025-02-07 05:51:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:03 UTC)" (scheduled at 2025-02-07 05:51:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:51:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:51:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 3, 419816), True, datetime.datetime(2025, 2, 7, 5, 51, 3, 419816), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:51:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:51:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:51:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:51:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:51:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:33.919286+00:00 (in 29.999558 seconds)
buildlogsarchiver stdout | 2025-02-07 05:51:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:33 UTC)" (scheduled at 2025-02-07 05:51:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:51:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 51, 3, 919974), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:51:03,926 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:51:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:51:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:51:04,875 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:51:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:51:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:59.937899+00:00 (in 55.000780 seconds)
exportactionlogsworker stdout | 2025-02-07 05:51:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:04 UTC)" (scheduled at 2025-02-07 05:51:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:51:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:51:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 4, 937401), True, datetime.datetime(2025, 2, 7, 5, 51, 4, 937401), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:51:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:51:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:51:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:51:06,012 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:51:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:51:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:18.732147+00:00 (in 9.999563 seconds)
notificationworker stdout | 2025-02-07 05:51:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:18 UTC)" (scheduled at 2025-02-07 05:51:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:51:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:51:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 8, 732890), True, datetime.datetime(2025, 2, 7, 5, 51, 8, 732890), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:51:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:51:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:51:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:51:10,303 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:51:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:51:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:24.423055+00:00 (in 13.998378 seconds)
gcworker stdout | 2025-02-07 05:51:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:40 UTC)" (scheduled at 2025-02-07 05:51:10.424250+00:00)
gcworker stdout | 2025-02-07 05:51:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:51:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907170430, None, 1, 0])
gcworker stdout | 2025-02-07 05:51:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:51:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:51:11,343 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:51:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:51:11,832 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:41.831521+00:00 (in 29.999528 seconds)
autopruneworker stdout | 2025-02-07 05:51:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:41 UTC)" (scheduled at 2025-02-07 05:51:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:51:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903871836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:51:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:51:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:51:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:51:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:51:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:29.225378+00:00 (in 14.999581 seconds)
quotatotalworker stdout | 2025-02-07 05:51:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:29 UTC)" (scheduled at 2025-02-07 05:51:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:51:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:51:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:51:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:51:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:51:14,236 [235] [DEBUG] [app] Starting request: urn:request:42a859e4-7e18-4cba-a027-1a3bb115d97a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:14,236 [237] [DEBUG] [app] Starting request: urn:request:0f562d69-a416-4867-a7e6-859e677f4c21 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:51:14,238 [244] [DEBUG] [app] Starting request: urn:request:c6769778-04bc-4ec5-a306-57ec5d3ab5d8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:14,238 [245] [DEBUG] [app] Starting request: urn:request:f103796d-773b-47fd-9883-8d5425540ab8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:14,239 [244] [DEBUG] [app] Ending request: urn:request:c6769778-04bc-4ec5-a306-57ec5d3ab5d8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c6769778-04bc-4ec5-a306-57ec5d3ab5d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:51:14,239 [245] [DEBUG] [app] Ending request: urn:request:f103796d-773b-47fd-9883-8d5425540ab8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f103796d-773b-47fd-9883-8d5425540ab8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:51:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:51:14,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:14,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:14,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:14,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:14,242 [237] [DEBUG] [app] Starting request: urn:request:74e31d04-608f-4c62-a67b-25ebdd0be89d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:51:14,242 [237] [DEBUG] [app] Ending request: urn:request:74e31d04-608f-4c62-a67b-25ebdd0be89d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:74e31d04-608f-4c62-a67b-25ebdd0be89d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:51:14,242 [236] [DEBUG] [app] Starting request: urn:request:5e9f0b5d-47c4-4f3b-afbb-bbcaf06fc605 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:51:14,242 [236] [DEBUG] [app] Ending request: urn:request:5e9f0b5d-47c4-4f3b-afbb-bbcaf06fc605 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5e9f0b5d-47c4-4f3b-afbb-bbcaf06fc605', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:51:14,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:14,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:14,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:14,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:14,243 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:14,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:14,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:14,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:14,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:14,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:14,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:14,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:14,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:14,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:14,253 [237] [DEBUG] [app] Ending request: urn:request:0f562d69-a416-4867-a7e6-859e677f4c21 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f562d69-a416-4867-a7e6-859e677f4c21', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:14,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:51:14,253 [235] [DEBUG] [app] Ending request: urn:request:42a859e4-7e18-4cba-a027-1a3bb115d97a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42a859e4-7e18-4cba-a027-1a3bb115d97a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:14,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:51:14,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 05:51:14,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
quotatotalworker stdout | 2025-02-07 05:51:15,388 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:51:15,932 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:51:16,432 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:51:17,868 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:51:18,117 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:51:18,120 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:51:18,122 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:51:18,127 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:51:18,130 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:51:18,618 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:51:18,659 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:51:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:51:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:28.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 05:51:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:28 UTC)" (scheduled at 2025-02-07 05:51:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:51:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:51:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 18, 732873), True, datetime.datetime(2025, 2, 7, 5, 51, 18, 732873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:51:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:51:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:51:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:51:19,125 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:51:19,127 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:51:19,130 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:51:19,992 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:20,218 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
servicekey stdout | 2025-02-07 05:51:20,614 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:51:20,620 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:51:20,626 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:51:20,630 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:51:20,632 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:51:20,635 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:51:20,639 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:51:20,641 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:51:20,644 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:51:20,651 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
globalpromstats stdout | 2025-02-07 05:51:20,783 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:51:20,984 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:51:21,387 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:51:21,761 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:51:22,062 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:51:22,113 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
repositorygcworker stdout | 2025-02-07 05:51:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:51:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:03.419110+00:00 (in 39.997717 seconds)
repositorygcworker stdout | 2025-02-07 05:51:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:23 UTC)" (scheduled at 2025-02-07 05:51:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:51:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:51:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:51:23,824 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:51:23,863 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:51:24,278 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:51:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:51:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:40.424250+00:00 (in 16.000719 seconds)
gcworker stdout | 2025-02-07 05:51:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:54 UTC)" (scheduled at 2025-02-07 05:51:24.423055+00:00)
gcworker stdout | 2025-02-07 05:51:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:51:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:51:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:51:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:51:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:38.732147+00:00 (in 9.999578 seconds)
notificationworker stdout | 2025-02-07 05:51:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:38 UTC)" (scheduled at 2025-02-07 05:51:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:51:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:51:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 28, 732849), True, datetime.datetime(2025, 2, 7, 5, 51, 28, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:51:28,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:51:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:51:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:51:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:51:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:44.225378+00:00 (in 14.999571 seconds)
quotatotalworker stdout | 2025-02-07 05:51:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:44 UTC)" (scheduled at 2025-02-07 05:51:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:51:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:51:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:51:29,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:51:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:51:29,235 [238] [DEBUG] [app] Starting request: urn:request:f467a557-58bb-4a90-b9f5-ba8e3ea686d9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:29,235 [235] [DEBUG] [app] Starting request: urn:request:36945685-3978-4f9c-838b-628c44f4a857 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:51:29,238 [245] [DEBUG] [app] Starting request: urn:request:56098089-9f51-4422-a4de-7e8372d588f4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:29,238 [239] [DEBUG] [app] Starting request: urn:request:4e547210-d85c-47ce-8fb7-fe17d64d2370 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:29,239 [245] [DEBUG] [app] Ending request: urn:request:56098089-9f51-4422-a4de-7e8372d588f4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56098089-9f51-4422-a4de-7e8372d588f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:51:29,239 [239] [DEBUG] [app] Ending request: urn:request:4e547210-d85c-47ce-8fb7-fe17d64d2370 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4e547210-d85c-47ce-8fb7-fe17d64d2370', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:51:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:51:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:29,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:29,242 [235] [DEBUG] [app] Starting request: urn:request:598e9321-c6b8-4f85-9168-ef44cb33058b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:51:29,242 [235] [DEBUG] [app] Ending request: urn:request:598e9321-c6b8-4f85-9168-ef44cb33058b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:598e9321-c6b8-4f85-9168-ef44cb33058b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:51:29,242 [237] [DEBUG] [app] Starting request: urn:request:a9973e2a-61b8-4d14-9c11-c0b5d34c43fb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:51:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:29,242 [237] [DEBUG] [app] Ending request: urn:request:a9973e2a-61b8-4d14-9c11-c0b5d34c43fb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9973e2a-61b8-4d14-9c11-c0b5d34c43fb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:29,253 [235] [DEBUG] [app] Ending request: urn:request:36945685-3978-4f9c-838b-628c44f4a857 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:36945685-3978-4f9c-838b-628c44f4a857', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:29,253 [238] [DEBUG] [app] Ending request: urn:request:f467a557-58bb-4a90-b9f5-ba8e3ea686d9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f467a557-58bb-4a90-b9f5-ba8e3ea686d9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:51:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:51:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:51:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
namespacegcworker stdout | 2025-02-07 05:51:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:51:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:32.625945+00:00 (in 1.004382 seconds)
namespacegcworker stdout | 2025-02-07 05:51:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:31 UTC)" (scheduled at 2025-02-07 05:51:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:51:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:51:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 31, 621820), True, datetime.datetime(2025, 2, 7, 5, 51, 31, 621820), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:51:31,628 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:51:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:51:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:51:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:51:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:13.624301+00:00 (in 40.997921 seconds)
namespacegcworker stdout | 2025-02-07 05:51:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:32 UTC)" (scheduled at 2025-02-07 05:51:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:51:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:51:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:51:33,254 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:51:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:51:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:03.919286+00:00 (in 29.999529 seconds)
buildlogsarchiver stdout | 2025-02-07 05:51:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:03 UTC)" (scheduled at 2025-02-07 05:51:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:51:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 51, 33, 920016), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:51:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:51:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:51:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:51:34,912 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:51:36,025 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:56.713547+00:00 (in 18.997222 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:37 UTC)" (scheduled at 2025-02-07 05:51:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:51:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:51:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:48.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:51:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:48 UTC)" (scheduled at 2025-02-07 05:51:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:51:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:51:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 38, 732846), True, datetime.datetime(2025, 2, 7, 5, 51, 38, 732846), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:51:38,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:51:38,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:51:38,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:51:40,331 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:51:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:51:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:54.423055+00:00 (in 13.998382 seconds)
gcworker stdout | 2025-02-07 05:51:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:10 UTC)" (scheduled at 2025-02-07 05:51:40.424250+00:00)
gcworker stdout | 2025-02-07 05:51:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:51:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907200431, None, 1, 0])
gcworker stdout | 2025-02-07 05:51:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:51:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:51:41,369 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:51:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:51:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:11.831521+00:00 (in 29.999559 seconds)
autopruneworker stdout | 2025-02-07 05:51:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:11 UTC)" (scheduled at 2025-02-07 05:51:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:51:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903901836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:51:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:51:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:51:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:51:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:51:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:59.225378+00:00 (in 14.999576 seconds)
quotatotalworker stdout | 2025-02-07 05:51:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:59 UTC)" (scheduled at 2025-02-07 05:51:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:51:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:51:44,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:51:44,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:51:44,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:51:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:51:44,235 [238] [DEBUG] [app] Starting request: urn:request:3313c97c-9033-4885-8222-e364658c9582 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:44,235 [235] [DEBUG] [app] Starting request: urn:request:9b54dc50-5c3b-4bfd-9d3d-b62301dc5d06 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:51:44,238 [245] [DEBUG] [app] Starting request: urn:request:265e434d-e130-4f70-991e-d29ad3566fe3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:44,238 [242] [DEBUG] [app] Starting request: urn:request:b93df19f-b9a7-4a02-bacc-4d91d948b1c0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:44,238 [242] [DEBUG] [app] Ending request: urn:request:b93df19f-b9a7-4a02-bacc-4d91d948b1c0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b93df19f-b9a7-4a02-bacc-4d91d948b1c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:51:44,238 [245] [DEBUG] [app] Ending request: urn:request:265e434d-e130-4f70-991e-d29ad3566fe3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:265e434d-e130-4f70-991e-d29ad3566fe3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:51:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:51:44,239 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:44,241 [237] [DEBUG] [app] Starting request: urn:request:36fea726-7da4-4a15-a4cc-27216aedbded (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:51:44,242 [237] [DEBUG] [app] Ending request: urn:request:36fea726-7da4-4a15-a4cc-27216aedbded (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36fea726-7da4-4a15-a4cc-27216aedbded', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:51:44,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:44,242 [236] [DEBUG] [app] Starting request: urn:request:4f3180b2-b503-47ad-8f1c-b8f467b50642 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:44,242 [236] [DEBUG] [app] Ending request: urn:request:4f3180b2-b503-47ad-8f1c-b8f467b50642 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f3180b2-b503-47ad-8f1c-b8f467b50642', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:51:44,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:44,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:44,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:44,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:44,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:44,253 [238] [DEBUG] [app] Ending request: urn:request:3313c97c-9033-4885-8222-e364658c9582 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3313c97c-9033-4885-8222-e364658c9582', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:44,253 [235] [DEBUG] [app] Ending request: urn:request:9b54dc50-5c3b-4bfd-9d3d-b62301dc5d06 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b54dc50-5c3b-4bfd-9d3d-b62301dc5d06', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:51:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:51:44,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:51:44,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:51:45,409 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:51:45,965 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:51:46,467 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:51:47,905 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:51:48,125 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:51:48,128 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:51:48,130 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:51:48,134 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:51:48,136 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:51:48,654 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:51:48,688 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:51:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:51:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:50.814677+00:00 (in 2.082098 seconds)
notificationworker stdout | 2025-02-07 05:51:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:58 UTC)" (scheduled at 2025-02-07 05:51:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:51:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:51:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 48, 732854), True, datetime.datetime(2025, 2, 7, 5, 51, 48, 732854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:51:48,742 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:51:48,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:51:48,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:51:58 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:51:49,133 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:51:49,136 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:51:49,139 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:51:50,012 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:50,254 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-registry stdout | 2025-02-07 05:51:50,631 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:51:50,634 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:51:50,637 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:51:50,639 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:51:50,642 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:51:50,645 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:51:50,649 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
servicekey stdout | 2025-02-07 05:51:50,652 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
gunicorn-registry stdout | 2025-02-07 05:51:50,655 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:51:50,661 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
notificationworker stdout | 2025-02-07 05:51:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:51:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:51:58.732147+00:00 (in 7.917040 seconds)
notificationworker stdout | 2025-02-07 05:51:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:50 UTC)" (scheduled at 2025-02-07 05:51:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:51:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:51:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:50 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:51:50,818 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:51:51,021 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:51:51,418 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:51:51,798 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:51:52,076 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:51:52,150 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
buildlogsarchiver stdout | 2025-02-07 05:51:53,843 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:51:53,899 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:51:54,315 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:51:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:51:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:10.424250+00:00 (in 16.000718 seconds)
gcworker stdout | 2025-02-07 05:51:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:24 UTC)" (scheduled at 2025-02-07 05:51:54.423055+00:00)
gcworker stdout | 2025-02-07 05:51:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:51:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:51:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:51:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:56.118855+00:00 (in 59.999528 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:51:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:56 UTC)" (scheduled at 2025-02-07 05:51:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:51:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:51:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:51:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:37.715878+00:00 (in 41.001914 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:56 UTC)" (scheduled at 2025-02-07 05:51:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:51:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:51:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:51:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:08.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:51:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:08 UTC)" (scheduled at 2025-02-07 05:51:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:51:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:51:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 51, 58, 732848), True, datetime.datetime(2025, 2, 7, 5, 51, 58, 732848), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:51:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:51:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:51:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:51:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:51:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:14.225378+00:00 (in 14.999538 seconds)
quotatotalworker stdout | 2025-02-07 05:51:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:14 UTC)" (scheduled at 2025-02-07 05:51:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:51:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:51:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:51:59,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:51:59,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:51:59,235 [237] [DEBUG] [app] Starting request: urn:request:2cf21bef-f481-41a0-b225-17f6848be548 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:59,235 [235] [DEBUG] [app] Starting request: urn:request:e0810c74-0929-4108-a430-0209e0a586c1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:51:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:51:59,238 [239] [DEBUG] [app] Starting request: urn:request:9eb6a806-c168-4561-a560-74d26c17be63 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:59,238 [245] [DEBUG] [app] Starting request: urn:request:3034b549-edf1-4b28-9d7d-84f90d3f1c59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:51:59,238 [239] [DEBUG] [app] Ending request: urn:request:9eb6a806-c168-4561-a560-74d26c17be63 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9eb6a806-c168-4561-a560-74d26c17be63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:51:59,239 [245] [DEBUG] [app] Ending request: urn:request:3034b549-edf1-4b28-9d7d-84f90d3f1c59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3034b549-edf1-4b28-9d7d-84f90d3f1c59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:51:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:51:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:59,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:59,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:51:59,241 [237] [DEBUG] [app] Starting request: urn:request:9977ea1d-87a7-4356-aa18-bb0101894b2a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:51:59,241 [237] [DEBUG] [app] Ending request: urn:request:9977ea1d-87a7-4356-aa18-bb0101894b2a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9977ea1d-87a7-4356-aa18-bb0101894b2a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:51:59,241 [238] [DEBUG] [app] Starting request: urn:request:bc7296b2-02aa-42c8-9459-2f8296ec8234 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:51:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:59,242 [238] [DEBUG] [app] Ending request: urn:request:bc7296b2-02aa-42c8-9459-2f8296ec8234 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc7296b2-02aa-42c8-9459-2f8296ec8234', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:59,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:51:59,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:51:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:51:59,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:51:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:51:59,242 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:51:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:51:59,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:59,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:51:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:51:59,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:59,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:51:59,251 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:59,251 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:51:59,252 [237] [DEBUG] [app] Ending request: urn:request:2cf21bef-f481-41a0-b225-17f6848be548 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2cf21bef-f481-41a0-b225-17f6848be548', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:59,252 [235] [DEBUG] [app] Ending request: urn:request:e0810c74-0929-4108-a430-0209e0a586c1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0810c74-0929-4108-a430-0209e0a586c1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:51:59,252 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:51:59,252 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:51:59,253 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:51:59,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:51:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:51:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.018)
exportactionlogsworker stdout | 2025-02-07 05:51:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:51:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:04.936688+00:00 (in 4.998306 seconds)
exportactionlogsworker stdout | 2025-02-07 05:51:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:59 UTC)" (scheduled at 2025-02-07 05:51:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:51:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:51:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:52:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:52:03,288 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:52:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:52:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:23.420886+00:00 (in 20.001296 seconds)
repositorygcworker stdout | 2025-02-07 05:52:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:03 UTC)" (scheduled at 2025-02-07 05:52:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:52:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:52:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 3, 419854), True, datetime.datetime(2025, 2, 7, 5, 52, 3, 419854), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:52:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:52:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:52:03,426 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:52:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:52:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:33.919286+00:00 (in 29.999513 seconds)
buildlogsarchiver stdout | 2025-02-07 05:52:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:33 UTC)" (scheduled at 2025-02-07 05:52:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:52:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 52, 3, 920059), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:52:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:52:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:52:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:33 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:52:04,932 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
exportactionlogsworker stdout | 2025-02-07 05:52:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:52:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:59.937899+00:00 (in 55.000788 seconds)
exportactionlogsworker stdout | 2025-02-07 05:52:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:04 UTC)" (scheduled at 2025-02-07 05:52:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:52:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:52:04,937 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 4, 937341), True, datetime.datetime(2025, 2, 7, 5, 52, 4, 937341), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:52:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:52:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:52:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:04 UTC)" executed successfully
queuecleanupworker stdout | 2025-02-07 05:52:06,043 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:52:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:18.732147+00:00 (in 9.999558 seconds)
notificationworker stdout | 2025-02-07 05:52:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:18 UTC)" (scheduled at 2025-02-07 05:52:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:52:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:52:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 8, 732870), True, datetime.datetime(2025, 2, 7, 5, 52, 8, 732870), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:52:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:52:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:52:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:52:10,356 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:52:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:52:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:24.423055+00:00 (in 13.998321 seconds)
gcworker stdout | 2025-02-07 05:52:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:40 UTC)" (scheduled at 2025-02-07 05:52:10.424250+00:00)
gcworker stdout | 2025-02-07 05:52:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:52:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907230431, None, 1, 0])
gcworker stdout | 2025-02-07 05:52:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:52:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:52:11,406 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:52:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:52:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:41.831521+00:00 (in 29.999550 seconds)
autopruneworker stdout | 2025-02-07 05:52:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:41 UTC)" (scheduled at 2025-02-07 05:52:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:52:11,837 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903931836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:52:11,840 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:52:11,840 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:52:11,840 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:41 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:52:13,624 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:52:13,624 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:31.621131+00:00 (in 17.996406 seconds)
namespacegcworker stdout | 2025-02-07 05:52:13,624 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:57:13 UTC)" (scheduled at 2025-02-07 05:52:13.624301+00:00)
namespacegcworker stdout | 2025-02-07 05:52:13,625 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 52, 13, 624972), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:52:13,631 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 52, 13, 624972), True, datetime.datetime(2025, 2, 7, 5, 52, 13, 624972), 0, 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:52:13,633 [71] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 52, 13, 624972), True, datetime.datetime(2025, 2, 7, 5, 52, 13, 624972), 0, 'namespacegc/%', False, datetime.datetime(2025, 2, 7, 5, 52, 13, 624972), 'namespacegc/%'])
namespacegcworker stdout | 2025-02-07 05:52:13,635 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:52:13,635 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:57:13 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:52:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:52:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:29.225378+00:00 (in 14.999487 seconds)
quotatotalworker stdout | 2025-02-07 05:52:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:29 UTC)" (scheduled at 2025-02-07 05:52:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:52:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:52:14,227 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:52:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:52:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:52:14,236 [237] [DEBUG] [app] Starting request: urn:request:744515c6-0dd8-426a-af81-465819c73409 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:14,236 [235] [DEBUG] [app] Starting request: urn:request:ca90df45-63c3-4ac2-97ed-d6add5173c5d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:52:14,239 [246] [DEBUG] [app] Starting request: urn:request:bc0b619b-32fa-4d35-b927-a4bb49bf5c4f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:14,239 [245] [DEBUG] [app] Starting request: urn:request:eb25340b-f4c9-4f16-bc43-11135429ec8e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:14,239 [245] [DEBUG] [app] Ending request: urn:request:eb25340b-f4c9-4f16-bc43-11135429ec8e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb25340b-f4c9-4f16-bc43-11135429ec8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:52:14,239 [246] [DEBUG] [app] Ending request: urn:request:bc0b619b-32fa-4d35-b927-a4bb49bf5c4f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bc0b619b-32fa-4d35-b927-a4bb49bf5c4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:52:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:52:14,240 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:52:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:14,242 [237] [DEBUG] [app] Starting request: urn:request:e213ec44-356c-443b-9f56-2430891c5ded (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:52:14,243 [236] [DEBUG] [app] Starting request: urn:request:da847d3c-0bdc-4033-a65c-24ec650a1018 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:52:14,243 [237] [DEBUG] [app] Ending request: urn:request:e213ec44-356c-443b-9f56-2430891c5ded (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e213ec44-356c-443b-9f56-2430891c5ded', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:52:14,243 [236] [DEBUG] [app] Ending request: urn:request:da847d3c-0bdc-4033-a65c-24ec650a1018 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:da847d3c-0bdc-4033-a65c-24ec650a1018', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:52:14,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:52:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:52:14,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:14,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:14,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:14,254 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:14,255 [237] [DEBUG] [app] Ending request: urn:request:744515c6-0dd8-426a-af81-465819c73409 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:744515c6-0dd8-426a-af81-465819c73409', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:14,255 [235] [DEBUG] [app] Ending request: urn:request:ca90df45-63c3-4ac2-97ed-d6add5173c5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca90df45-63c3-4ac2-97ed-d6add5173c5d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:52:14,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:52:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:52:14,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:52:15,442 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:52:16,001 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:52:16,499 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:52:17,942 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:52:18,133 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:52:18,136 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:52:18,139 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:52:18,142 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:52:18,145 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:52:18,690 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:52:18,724 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
notificationworker stdout | 2025-02-07 05:52:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:28.732147+00:00 (in 9.999606 seconds)
notificationworker stdout | 2025-02-07 05:52:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:28 UTC)" (scheduled at 2025-02-07 05:52:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:52:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:52:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 18, 732855), True, datetime.datetime(2025, 2, 7, 5, 52, 18, 732855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:52:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:52:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:52:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:28 UTC)" executed successfully
gunicorn-secscan stdout | 2025-02-07 05:52:19,141 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:52:19,144 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:52:19,146 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:52:20,027 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:20,264 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-registry stdout | 2025-02-07 05:52:20,642 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:52:20,645 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:52:20,648 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:52:20,652 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:52:20,655 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:52:20,657 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:52:20,661 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:52:20,666 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:52:20,672 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
servicekey stdout | 2025-02-07 05:52:20,689 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:52:20,831 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:52:21,057 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:52:21,444 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:52:21,834 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:52:22,090 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:52:22,175 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
repositorygcworker stdout | 2025-02-07 05:52:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:52:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:03.419110+00:00 (in 39.997783 seconds)
repositorygcworker stdout | 2025-02-07 05:52:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:23 UTC)" (scheduled at 2025-02-07 05:52:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:52:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:52:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:52:23,853 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:52:23,906 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:52:24,343 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:52:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:52:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:40.424250+00:00 (in 16.000743 seconds)
gcworker stdout | 2025-02-07 05:52:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:54 UTC)" (scheduled at 2025-02-07 05:52:24.423055+00:00)
gcworker stdout | 2025-02-07 05:52:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:52:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:52:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:52:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:33.814166+00:00 (in 5.081630 seconds)
notificationworker stdout | 2025-02-07 05:52:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:38 UTC)" (scheduled at 2025-02-07 05:52:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:52:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:52:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 28, 732814), True, datetime.datetime(2025, 2, 7, 5, 52, 28, 732814), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:52:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:52:28,742 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:52:28,742 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:52:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:52:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:44.225378+00:00 (in 14.999561 seconds)
quotatotalworker stdout | 2025-02-07 05:52:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:44 UTC)" (scheduled at 2025-02-07 05:52:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:52:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:52:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:52:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:52:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:52:29,235 [235] [DEBUG] [app] Starting request: urn:request:bed9f9df-b801-4aab-8293-8e99010cfb74 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:29,235 [238] [DEBUG] [app] Starting request: urn:request:b4d550ea-b4e2-4cd4-b3f7-aa2775cf1f19 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:52:29,238 [245] [DEBUG] [app] Starting request: urn:request:583d20b4-bc2b-4a50-9711-31ff11bc6c18 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:29,238 [239] [DEBUG] [app] Starting request: urn:request:c2e801aa-d3df-436e-9798-38551846dbf5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:29,239 [245] [DEBUG] [app] Ending request: urn:request:583d20b4-bc2b-4a50-9711-31ff11bc6c18 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:583d20b4-bc2b-4a50-9711-31ff11bc6c18', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:52:29,239 [239] [DEBUG] [app] Ending request: urn:request:c2e801aa-d3df-436e-9798-38551846dbf5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c2e801aa-d3df-436e-9798-38551846dbf5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:52:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:52:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:52:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:29,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:29,241 [237] [DEBUG] [app] Starting request: urn:request:2f5d8946-86bd-4c17-b005-1b4cb7d10495 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:52:29,242 [237] [DEBUG] [app] Ending request: urn:request:2f5d8946-86bd-4c17-b005-1b4cb7d10495 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f5d8946-86bd-4c17-b005-1b4cb7d10495', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:52:29,242 [238] [DEBUG] [app] Starting request: urn:request:be9c0439-a54a-4e13-95f2-1544c47a2337 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:52:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:52:29,242 [238] [DEBUG] [app] Ending request: urn:request:be9c0439-a54a-4e13-95f2-1544c47a2337 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:be9c0439-a54a-4e13-95f2-1544c47a2337', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:52:29,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:52:29,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:29,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:29,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:29,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:29,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:29,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:29,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:29,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:29,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:29,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:29,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:29,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:29,254 [238] [DEBUG] [app] Ending request: urn:request:b4d550ea-b4e2-4cd4-b3f7-aa2775cf1f19 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b4d550ea-b4e2-4cd4-b3f7-aa2775cf1f19', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:29,254 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:52:29,254 [235] [DEBUG] [app] Ending request: urn:request:bed9f9df-b801-4aab-8293-8e99010cfb74 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bed9f9df-b801-4aab-8293-8e99010cfb74', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:29,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:52:29,254 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.019)
gunicorn-web stdout | 2025-02-07 05:52:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:52:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:52:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:32.625945+00:00 (in 1.004344 seconds)
namespacegcworker stdout | 2025-02-07 05:52:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:31 UTC)" (scheduled at 2025-02-07 05:52:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:52:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:52:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 31, 621906), True, datetime.datetime(2025, 2, 7, 5, 52, 31, 621906), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:52:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:52:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:52:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:52:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:52:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:31.621131+00:00 (in 58.994671 seconds)
namespacegcworker stdout | 2025-02-07 05:52:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:32 UTC)" (scheduled at 2025-02-07 05:52:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:52:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:52:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:52:33,318 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
notificationworker stdout | 2025-02-07 05:52:33,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:33,814 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:38.732147+00:00 (in 4.917559 seconds)
notificationworker stdout | 2025-02-07 05:52:33,814 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:57:33 UTC)" (scheduled at 2025-02-07 05:52:33.814166+00:00)
notificationworker stdout | 2025-02-07 05:52:33,815 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 52, 33, 814862), 'notification/%'])
notificationworker stdout | 2025-02-07 05:52:33,821 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 52, 33, 814862), True, datetime.datetime(2025, 2, 7, 5, 52, 33, 814862), 0, 'notification/%'])
notificationworker stdout | 2025-02-07 05:52:33,823 [73] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 52, 33, 814862), True, datetime.datetime(2025, 2, 7, 5, 52, 33, 814862), 0, 'notification/%', False, datetime.datetime(2025, 2, 7, 5, 52, 33, 814862), 'notification/%'])
notificationworker stdout | 2025-02-07 05:52:33,825 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:52:33,825 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:57:33 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:52:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:52:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:03.919286+00:00 (in 29.999497 seconds)
buildlogsarchiver stdout | 2025-02-07 05:52:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:03 UTC)" (scheduled at 2025-02-07 05:52:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:52:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 52, 33, 920040), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:52:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:52:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:52:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:52:34,969 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:52:36,079 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:56.713547+00:00 (in 18.997238 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:37 UTC)" (scheduled at 2025-02-07 05:52:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:37,716 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:52:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:48.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:52:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:48 UTC)" (scheduled at 2025-02-07 05:52:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:52:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:52:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 38, 732888), True, datetime.datetime(2025, 2, 7, 5, 52, 38, 732888), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:52:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:52:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:52:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:48 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:52:40,390 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:52:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:52:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:54.423055+00:00 (in 13.998384 seconds)
gcworker stdout | 2025-02-07 05:52:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:10 UTC)" (scheduled at 2025-02-07 05:52:40.424250+00:00)
gcworker stdout | 2025-02-07 05:52:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:52:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907260430, None, 1, 0])
gcworker stdout | 2025-02-07 05:52:40,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:52:40,432 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:10 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:52:41,415 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:52:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:52:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:11.831521+00:00 (in 29.999546 seconds)
autopruneworker stdout | 2025-02-07 05:52:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:11 UTC)" (scheduled at 2025-02-07 05:52:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:52:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903961836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:52:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:52:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:52:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:52:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:52:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:59.225378+00:00 (in 14.999565 seconds)
quotatotalworker stdout | 2025-02-07 05:52:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:59 UTC)" (scheduled at 2025-02-07 05:52:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:52:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:52:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:52:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:52:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:52:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:52:44,235 [238] [DEBUG] [app] Starting request: urn:request:df01ce39-c90a-46e6-a85e-7a9da9eac97d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:44,235 [235] [DEBUG] [app] Starting request: urn:request:b3fbf4d5-a0ba-4141-af40-87420298e0c3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:44,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:52:44,238 [244] [DEBUG] [app] Starting request: urn:request:d923c774-dfef-4cda-8bd1-5c94dc2abac7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:44,239 [239] [DEBUG] [app] Starting request: urn:request:a2744090-38de-4075-b755-7f92d313cabe (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:44,239 [244] [DEBUG] [app] Ending request: urn:request:d923c774-dfef-4cda-8bd1-5c94dc2abac7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d923c774-dfef-4cda-8bd1-5c94dc2abac7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:52:44,239 [239] [DEBUG] [app] Ending request: urn:request:a2744090-38de-4075-b755-7f92d313cabe (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2744090-38de-4075-b755-7f92d313cabe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:52:44,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:52:44,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:52:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:44,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:44,242 [238] [DEBUG] [app] Starting request: urn:request:3774aea5-390e-4302-a7fa-1dbc23c4c276 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:52:44,242 [238] [DEBUG] [app] Ending request: urn:request:3774aea5-390e-4302-a7fa-1dbc23c4c276 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3774aea5-390e-4302-a7fa-1dbc23c4c276', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:52:44,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:52:44,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:44,243 [237] [DEBUG] [app] Starting request: urn:request:86fc6d1f-fc28-4a7c-a133-ae40bdcc7096 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:52:44,243 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:44,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:44,243 [237] [DEBUG] [app] Ending request: urn:request:86fc6d1f-fc28-4a7c-a133-ae40bdcc7096 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:86fc6d1f-fc28-4a7c-a133-ae40bdcc7096', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:52:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:52:44,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:44,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:44,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:44,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:44,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:44,251 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:44,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:44,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:44,253 [238] [DEBUG] [app] Ending request: urn:request:df01ce39-c90a-46e6-a85e-7a9da9eac97d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df01ce39-c90a-46e6-a85e-7a9da9eac97d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:44,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:52:44,254 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:52:44,254 [235] [DEBUG] [app] Ending request: urn:request:b3fbf4d5-a0ba-4141-af40-87420298e0c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b3fbf4d5-a0ba-4141-af40-87420298e0c3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:44,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:52:44,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:52:45,478 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:52:46,038 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:52:46,506 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:52:47,968 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:52:48,140 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:52:48,143 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:52:48,146 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:52:48,149 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:52:48,152 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:52:48,717 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
notificationworker stdout | 2025-02-07 05:52:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:50.814677+00:00 (in 2.082164 seconds)
notificationworker stdout | 2025-02-07 05:52:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:58 UTC)" (scheduled at 2025-02-07 05:52:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:52:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:52:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 48, 732775), True, datetime.datetime(2025, 2, 7, 5, 52, 48, 732775), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:52:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:52:48,739 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:52:48,739 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:52:58 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 05:52:48,747 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-secscan stdout | 2025-02-07 05:52:49,149 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:52:49,152 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:52:49,155 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:52:50,063 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:50,289 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-registry stdout | 2025-02-07 05:52:50,651 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:52:50,653 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:52:50,656 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:52:50,662 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:52:50,665 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:52:50,668 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:52:50,671 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:52:50,675 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:52:50,680 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
servicekey stdout | 2025-02-07 05:52:50,717 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:52:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:52:58.732147+00:00 (in 7.917048 seconds)
notificationworker stdout | 2025-02-07 05:52:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:50 UTC)" (scheduled at 2025-02-07 05:52:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:52:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:52:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:50 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:52:50,852 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:52:51,093 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:52:51,481 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:52:51,870 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:52:52,126 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:52:52,199 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
buildlogsarchiver stdout | 2025-02-07 05:52:53,889 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:52:53,928 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:52:54,379 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:52:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:52:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:10.424250+00:00 (in 16.000765 seconds)
gcworker stdout | 2025-02-07 05:52:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:24 UTC)" (scheduled at 2025-02-07 05:52:54.423055+00:00)
gcworker stdout | 2025-02-07 05:52:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:52:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:24 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:52:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:52:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:56.118855+00:00 (in 59.999511 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:52:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:56 UTC)" (scheduled at 2025-02-07 05:52:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:52:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:52:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:52:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:56,713 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:37.715878+00:00 (in 41.001902 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:56 UTC)" (scheduled at 2025-02-07 05:52:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:52:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:52:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:52:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:08.732147+00:00 (in 9.999537 seconds)
notificationworker stdout | 2025-02-07 05:52:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:08 UTC)" (scheduled at 2025-02-07 05:52:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:52:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:52:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 52, 58, 732943), True, datetime.datetime(2025, 2, 7, 5, 52, 58, 732943), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:52:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:52:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:52:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:52:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:52:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:14.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 05:52:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:14 UTC)" (scheduled at 2025-02-07 05:52:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:52:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:52:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:52:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:52:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:52:59,235 [237] [DEBUG] [app] Starting request: urn:request:47bdbce5-7b6d-43ed-bc5c-c212a9785d4a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:59,235 [235] [DEBUG] [app] Starting request: urn:request:28ba7b4a-287b-47fb-bdd6-40ae67bad4d4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:52:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:59,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:52:59,239 [239] [DEBUG] [app] Starting request: urn:request:2aac8408-a119-4a11-97f4-e9e28edeb488 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:59,239 [245] [DEBUG] [app] Starting request: urn:request:bb256a9c-5700-49ec-9f86-1405aaa11bd7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:52:59,239 [239] [DEBUG] [app] Ending request: urn:request:2aac8408-a119-4a11-97f4-e9e28edeb488 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2aac8408-a119-4a11-97f4-e9e28edeb488', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:52:59,239 [245] [DEBUG] [app] Ending request: urn:request:bb256a9c-5700-49ec-9f86-1405aaa11bd7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bb256a9c-5700-49ec-9f86-1405aaa11bd7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:52:59,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:52:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:52:59,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:59,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:59,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:59,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:52:59,242 [237] [DEBUG] [app] Starting request: urn:request:0cd024f7-9cfa-419d-a285-c24a418c9b38 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:52:59,243 [237] [DEBUG] [app] Ending request: urn:request:0cd024f7-9cfa-419d-a285-c24a418c9b38 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0cd024f7-9cfa-419d-a285-c24a418c9b38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:52:59,243 [238] [DEBUG] [app] Starting request: urn:request:fc7be223-7794-4612-bf79-589dca7d0633 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:52:59,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:52:59,243 [238] [DEBUG] [app] Ending request: urn:request:fc7be223-7794-4612-bf79-589dca7d0633 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc7be223-7794-4612-bf79-589dca7d0633', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:52:59,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:52:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:52:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:52:59,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:59,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:52:59,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:59,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:52:59,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:52:59,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:59,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:59,248 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:52:59,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:52:59,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:59,252 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:52:59,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:52:59,254 [237] [DEBUG] [app] Ending request: urn:request:47bdbce5-7b6d-43ed-bc5c-c212a9785d4a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47bdbce5-7b6d-43ed-bc5c-c212a9785d4a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:59,254 [235] [DEBUG] [app] Ending request: urn:request:28ba7b4a-287b-47fb-bdd6-40ae67bad4d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28ba7b4a-287b-47fb-bdd6-40ae67bad4d4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:52:59,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:52:59,255 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:52:59,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:52:59,255 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:52:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:52:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:52:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:52:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:04.936688+00:00 (in 4.998309 seconds)
exportactionlogsworker stdout | 2025-02-07 05:52:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:59 UTC)" (scheduled at 2025-02-07 05:52:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:52:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:52:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:53:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:53:03,354 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:53:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:53:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:23.420886+00:00 (in 20.001305 seconds)
repositorygcworker stdout | 2025-02-07 05:53:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:03 UTC)" (scheduled at 2025-02-07 05:53:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:53:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:53:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 3, 419895), True, datetime.datetime(2025, 2, 7, 5, 53, 3, 419895), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:53:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:53:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:53:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:53:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:53:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:33.919286+00:00 (in 29.999549 seconds)
buildlogsarchiver stdout | 2025-02-07 05:53:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:33 UTC)" (scheduled at 2025-02-07 05:53:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:53:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 53, 3, 920005), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:53:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:53:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:53:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:53:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:53:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:59.937899+00:00 (in 55.000740 seconds)
exportactionlogsworker stdout | 2025-02-07 05:53:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:04 UTC)" (scheduled at 2025-02-07 05:53:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:53:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:53:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 4, 937392), True, datetime.datetime(2025, 2, 7, 5, 53, 4, 937392), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:53:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:53:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:53:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:04 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:53:05,006 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:53:06,111 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:53:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:53:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:18.732147+00:00 (in 9.999568 seconds)
notificationworker stdout | 2025-02-07 05:53:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:18 UTC)" (scheduled at 2025-02-07 05:53:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:53:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:53:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 8, 732882), True, datetime.datetime(2025, 2, 7, 5, 53, 8, 732882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:53:08,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:53:08,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:53:08,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:18 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:53:10,397 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
gcworker stdout | 2025-02-07 05:53:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:53:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:24.423055+00:00 (in 13.998392 seconds)
gcworker stdout | 2025-02-07 05:53:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:40 UTC)" (scheduled at 2025-02-07 05:53:10.424250+00:00)
gcworker stdout | 2025-02-07 05:53:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:53:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907290431, None, 1, 0])
gcworker stdout | 2025-02-07 05:53:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:53:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:53:11,422 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:53:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:53:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:41.831521+00:00 (in 29.999535 seconds)
autopruneworker stdout | 2025-02-07 05:53:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:41 UTC)" (scheduled at 2025-02-07 05:53:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:53:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738903991836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:53:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:53:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:53:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:53:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:53:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:29.225378+00:00 (in 14.999568 seconds)
quotatotalworker stdout | 2025-02-07 05:53:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:29 UTC)" (scheduled at 2025-02-07 05:53:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:53:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:53:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:53:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:53:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:53:14,235 [238] [DEBUG] [app] Starting request: urn:request:e2832dcd-b00c-4748-b79a-dfd08267da06 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:14,235 [237] [DEBUG] [app] Starting request: urn:request:01332deb-274b-4234-8dab-b0bf7de3bc80 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:14,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:14,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:53:14,238 [244] [DEBUG] [app] Starting request: urn:request:932bf815-1c28-43a4-a0c1-86c5878b4fc8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:14,238 [239] [DEBUG] [app] Starting request: urn:request:88a88a01-18ba-476b-90ec-795bd709ab90 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:14,239 [239] [DEBUG] [app] Ending request: urn:request:88a88a01-18ba-476b-90ec-795bd709ab90 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:88a88a01-18ba-476b-90ec-795bd709ab90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:53:14,239 [244] [DEBUG] [app] Ending request: urn:request:932bf815-1c28-43a4-a0c1-86c5878b4fc8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:932bf815-1c28-43a4-a0c1-86c5878b4fc8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:53:14,239 [244] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:53:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:14,240 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:14,240 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:14,242 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:14,243 [236] [DEBUG] [app] Starting request: urn:request:3e638b9d-485f-4806-bd9e-51c678e38fa6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:14,243 [235] [DEBUG] [app] Starting request: urn:request:c38a4bbb-2f15-4ad0-98b0-f1769490034a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:14,243 [236] [DEBUG] [app] Ending request: urn:request:3e638b9d-485f-4806-bd9e-51c678e38fa6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3e638b9d-485f-4806-bd9e-51c678e38fa6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:53:14,244 [235] [DEBUG] [app] Ending request: urn:request:c38a4bbb-2f15-4ad0-98b0-f1769490034a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c38a4bbb-2f15-4ad0-98b0-f1769490034a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:53:14,244 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:53:14,244 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:14,244 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:14,244 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:14,249 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:14,253 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:14,253 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:14,254 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:14,254 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:14,255 [238] [DEBUG] [app] Ending request: urn:request:e2832dcd-b00c-4748-b79a-dfd08267da06 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e2832dcd-b00c-4748-b79a-dfd08267da06', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:53:14,255 [237] [DEBUG] [app] Ending request: urn:request:01332deb-274b-4234-8dab-b0bf7de3bc80 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01332deb-274b-4234-8dab-b0bf7de3bc80', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:53:14,256 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:53:14,256 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:53:14,256 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:53:14,256 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:53:15,514 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:53:16,074 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:53:16,514 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:53:18,005 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:53:18,148 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:53:18,151 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:53:18,154 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:53:18,156 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:53:18,160 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:53:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:53:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:28.732147+00:00 (in 9.999653 seconds)
notificationworker stdout | 2025-02-07 05:53:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:28 UTC)" (scheduled at 2025-02-07 05:53:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:53:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:53:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 18, 732752), True, datetime.datetime(2025, 2, 7, 5, 53, 18, 732752), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:53:18,735 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
notificationworker stdout | 2025-02-07 05:53:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:53:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:53:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:28 UTC)" executed successfully
chunkcleanupworker stdout | 2025-02-07 05:53:18,774 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-secscan stdout | 2025-02-07 05:53:19,157 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:53:19,160 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:53:19,163 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:53:20,100 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:20,325 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-registry stdout | 2025-02-07 05:53:20,659 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:53:20,664 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:53:20,667 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:53:20,672 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:53:20,676 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:53:20,678 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:53:20,682 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:53:20,685 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:53:20,689 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
servicekey stdout | 2025-02-07 05:53:20,736 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:53:20,887 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:53:21,129 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:53:21,516 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:53:21,887 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:53:22,140 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:53:22,235 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
repositorygcworker stdout | 2025-02-07 05:53:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:53:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:32.420707+00:00 (in 8.999340 seconds)
repositorygcworker stdout | 2025-02-07 05:53:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:23 UTC)" (scheduled at 2025-02-07 05:53:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:53:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:53:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:53:23,896 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:53:23,936 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
blobuploadcleanupworker stdout | 2025-02-07 05:53:24,415 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
gcworker stdout | 2025-02-07 05:53:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:53:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:40.424250+00:00 (in 16.000762 seconds)
gcworker stdout | 2025-02-07 05:53:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:54 UTC)" (scheduled at 2025-02-07 05:53:24.423055+00:00)
gcworker stdout | 2025-02-07 05:53:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:53:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:53:54 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:53:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:53:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:38.732147+00:00 (in 9.999579 seconds)
notificationworker stdout | 2025-02-07 05:53:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:38 UTC)" (scheduled at 2025-02-07 05:53:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:53:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:53:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 28, 732886), True, datetime.datetime(2025, 2, 7, 5, 53, 28, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:53:28,741 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:53:28,741 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:53:28,741 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:53:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:53:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:44.225378+00:00 (in 14.999560 seconds)
quotatotalworker stdout | 2025-02-07 05:53:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:44 UTC)" (scheduled at 2025-02-07 05:53:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:53:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:53:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:53:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:53:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:53:29,235 [235] [DEBUG] [app] Starting request: urn:request:04646049-5ac7-46e8-b7f1-03c3e4ec75fd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:29,235 [238] [DEBUG] [app] Starting request: urn:request:ca905a09-2c08-4f03-bfb4-2c7d66dfa526 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:29,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:29,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:53:29,238 [239] [DEBUG] [app] Starting request: urn:request:19204933-6f74-4ef3-ba53-8445c604723f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:29,238 [246] [DEBUG] [app] Starting request: urn:request:57735b3f-a987-4bb2-9d63-7c839ff5afec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:29,238 [246] [DEBUG] [app] Ending request: urn:request:57735b3f-a987-4bb2-9d63-7c839ff5afec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:57735b3f-a987-4bb2-9d63-7c839ff5afec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:53:29,238 [239] [DEBUG] [app] Ending request: urn:request:19204933-6f74-4ef3-ba53-8445c604723f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:19204933-6f74-4ef3-ba53-8445c604723f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:53:29,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:53:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:29,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:29,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:29,241 [238] [DEBUG] [app] Starting request: urn:request:52cb0d71-2112-4b2c-9f70-1eadc195175e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:29,241 [238] [DEBUG] [app] Ending request: urn:request:52cb0d71-2112-4b2c-9f70-1eadc195175e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:52cb0d71-2112-4b2c-9f70-1eadc195175e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:53:29,241 [237] [DEBUG] [app] Starting request: urn:request:f0a0ba56-f429-4c31-b91f-d3eff4562a6d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:29,242 [237] [DEBUG] [app] Ending request: urn:request:f0a0ba56-f429-4c31-b91f-d3eff4562a6d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f0a0ba56-f429-4c31-b91f-d3eff4562a6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:53:29,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:29,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:53:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:29,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:29,242 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:29,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:29,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:29,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:29,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:29,250 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:29,252 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:29,253 [238] [DEBUG] [app] Ending request: urn:request:ca905a09-2c08-4f03-bfb4-2c7d66dfa526 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca905a09-2c08-4f03-bfb4-2c7d66dfa526', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:53:29,253 [235] [DEBUG] [app] Ending request: urn:request:04646049-5ac7-46e8-b7f1-03c3e4ec75fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:04646049-5ac7-46e8-b7f1-03c3e4ec75fd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:53:29,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:53:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:53:29,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:53:29,253 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
namespacegcworker stdout | 2025-02-07 05:53:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:53:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:32.625945+00:00 (in 1.004314 seconds)
namespacegcworker stdout | 2025-02-07 05:53:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:31 UTC)" (scheduled at 2025-02-07 05:53:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:53:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:53:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 31, 621948), True, datetime.datetime(2025, 2, 7, 5, 53, 31, 621948), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:53:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:53:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:53:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:31 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:53:32,420 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:53:32,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:03.419110+00:00 (in 30.997989 seconds)
repositorygcworker stdout | 2025-02-07 05:53:32,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:58:32 UTC)" (scheduled at 2025-02-07 05:53:32.420707+00:00)
repositorygcworker stdout | 2025-02-07 05:53:32,421 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 53, 32, 421386), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:53:32,428 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 53, 32, 421386), True, datetime.datetime(2025, 2, 7, 5, 53, 32, 421386), 0, 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:53:32,429 [93] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 53, 32, 421386), True, datetime.datetime(2025, 2, 7, 5, 53, 32, 421386), 0, 'repositorygc/%', False, datetime.datetime(2025, 2, 7, 5, 53, 32, 421386), 'repositorygc/%'])
repositorygcworker stdout | 2025-02-07 05:53:32,431 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:53:32,431 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:58:32 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:53:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:53:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:31.621131+00:00 (in 58.994750 seconds)
namespacegcworker stdout | 2025-02-07 05:53:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:32 UTC)" (scheduled at 2025-02-07 05:53:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:53:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:53:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:53:33,390 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:53:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:53:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:03.919286+00:00 (in 29.999526 seconds)
buildlogsarchiver stdout | 2025-02-07 05:53:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:03 UTC)" (scheduled at 2025-02-07 05:53:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:53:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 53, 33, 920037), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:53:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:53:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:53:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:53:35,036 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:53:36,148 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:56.713547+00:00 (in 18.997227 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:37 UTC)" (scheduled at 2025-02-07 05:53:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:53:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:53:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:48.732147+00:00 (in 9.999576 seconds)
notificationworker stdout | 2025-02-07 05:53:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:48 UTC)" (scheduled at 2025-02-07 05:53:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:53:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:53:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 38, 732834), True, datetime.datetime(2025, 2, 7, 5, 53, 38, 732834), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:53:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:53:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:53:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:53:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:53:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:54.423055+00:00 (in 13.998374 seconds)
gcworker stdout | 2025-02-07 05:53:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:10 UTC)" (scheduled at 2025-02-07 05:53:40.424250+00:00)
gcworker stdout | 2025-02-07 05:53:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:53:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907320431, None, 1, 0])
gcworker stdout | 2025-02-07 05:53:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:53:40,434 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:10 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:53:40,434 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:53:41,434 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:53:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:53:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:11.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 05:53:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:11 UTC)" (scheduled at 2025-02-07 05:53:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:53:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738904021836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:53:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:53:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:53:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:53:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:53:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:59.225378+00:00 (in 14.999572 seconds)
quotatotalworker stdout | 2025-02-07 05:53:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:59 UTC)" (scheduled at 2025-02-07 05:53:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:53:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:53:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:53:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:53:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:53:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:53:44,235 [235] [DEBUG] [app] Starting request: urn:request:08786a7e-077c-4b5a-b20d-989bdb85c284 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:44,235 [238] [DEBUG] [app] Starting request: urn:request:77d6ba19-7ac9-4ce4-9f37-6cb5377dcfd9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:44,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:44,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:53:44,238 [245] [DEBUG] [app] Starting request: urn:request:e309cde3-f25f-42d7-a525-b677349e5008 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:44,238 [240] [DEBUG] [app] Starting request: urn:request:3baec915-b4ec-4732-adb6-0cf6bfeae48c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:44,238 [245] [DEBUG] [app] Ending request: urn:request:e309cde3-f25f-42d7-a525-b677349e5008 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e309cde3-f25f-42d7-a525-b677349e5008', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:53:44,238 [240] [DEBUG] [app] Ending request: urn:request:3baec915-b4ec-4732-adb6-0cf6bfeae48c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3baec915-b4ec-4732-adb6-0cf6bfeae48c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:53:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:53:44,239 [240] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:44,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:44,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:44,242 [236] [DEBUG] [app] Starting request: urn:request:16d26b93-5d15-45e8-9a4a-fcb25978b752 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:44,242 [237] [DEBUG] [app] Starting request: urn:request:74d50408-c2d0-49f3-859a-81d91b606df2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:44,243 [237] [DEBUG] [app] Ending request: urn:request:74d50408-c2d0-49f3-859a-81d91b606df2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:74d50408-c2d0-49f3-859a-81d91b606df2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:53:44,243 [236] [DEBUG] [app] Ending request: urn:request:16d26b93-5d15-45e8-9a4a-fcb25978b752 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16d26b93-5d15-45e8-9a4a-fcb25978b752', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:53:44,243 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:44,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:53:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:44,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:44,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:44,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:44,247 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:44,247 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:44,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:44,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:44,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:44,254 [235] [DEBUG] [app] Ending request: urn:request:08786a7e-077c-4b5a-b20d-989bdb85c284 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:08786a7e-077c-4b5a-b20d-989bdb85c284', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:53:44,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:53:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:53:44,254 [238] [DEBUG] [app] Ending request: urn:request:77d6ba19-7ac9-4ce4-9f37-6cb5377dcfd9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77d6ba19-7ac9-4ce4-9f37-6cb5377dcfd9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
gunicorn-web stdout | 2025-02-07 05:53:44,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:53:44,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:53:45,550 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:53:46,107 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:53:46,550 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:53:48,021 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:53:48,157 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:53:48,160 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:53:48,164 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:53:48,167 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:53:48,170 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:53:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:53:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:50.814677+00:00 (in 2.082074 seconds)
notificationworker stdout | 2025-02-07 05:53:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:58 UTC)" (scheduled at 2025-02-07 05:53:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:53:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:53:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 48, 732888), True, datetime.datetime(2025, 2, 7, 5, 53, 48, 732888), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:53:48,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:53:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:53:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:53:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:53:48,748 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:53:48,810 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-secscan stdout | 2025-02-07 05:53:49,168 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:53:49,172 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:53:49,175 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:53:50,122 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:50,356 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-registry stdout | 2025-02-07 05:53:50,666 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:53:50,675 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:53:50,679 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:53:50,682 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:53:50,685 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:53:50,688 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:53:50,690 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:53:50,693 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:53:50,697 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
servicekey stdout | 2025-02-07 05:53:50,747 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:53:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:53:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:53:58.732147+00:00 (in 7.917024 seconds)
notificationworker stdout | 2025-02-07 05:53:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:50 UTC)" (scheduled at 2025-02-07 05:53:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:53:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:53:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:50 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:53:50,906 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:53:51,142 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:53:51,544 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:53:51,902 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:53:52,149 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:53:52,271 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
buildlogsarchiver stdout | 2025-02-07 05:53:53,925 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:53:53,972 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gcworker stdout | 2025-02-07 05:53:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:53:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:10.424250+00:00 (in 16.000745 seconds)
gcworker stdout | 2025-02-07 05:53:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:24 UTC)" (scheduled at 2025-02-07 05:53:54.423055+00:00)
gcworker stdout | 2025-02-07 05:53:54,423 [62] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2025-02-07 05:53:54,429 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:53:54,429 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:53:54,430 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:24 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:53:54,451 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
servicekey stdout | 2025-02-07 05:53:55,325 [99] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2025-02-07 05:53:55,325 [99] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 06:48:55.325482+00:00 (in 3299.999547 seconds)
servicekey stdout | 2025-02-07 05:53:55,326 [99] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-02-07 06:48:55 UTC)" (scheduled at 2025-02-07 05:53:55.325482+00:00)
servicekey stdout | 2025-02-07 05:53:55,326 [99] [DEBUG] [__main__] Starting automatic refresh of service key rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4 to new expiration 2025-02-07 07:53:55.326120
servicekey stdout | 2025-02-07 05:53:55,326 [99] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4', datetime.datetime(2025, 1, 31, 5, 53, 55, 326296), None, 1, 0])
servicekey stdout | 2025-02-07 05:53:55,333 [99] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise-15131.apps.quaytest-15131.qe.devcluster.openshift.com:443', 'rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4', 'quay', '{"n": "ydZTwWkSP7GLo1VWTY2-tvY2zs-NGpukK9A_Pr1ftW_Qt6JvAWK7vQNC1EA6mULUpUnSEnza3ySALI1yWv9VMgurD8CDnre3MgeFTD3HgFNEnP5af62LG9xYf376yvB9fKjs0XIvd7kbOqdixWiBBR1AGkmOai4M3ehFLQ1vhqcdOlPSPvGgQsRwzTGY78K72BJvpsmB-7yGfu23ebGgLJPjKIwBoWFAsogIVyfxg3kF0x6hFdlu3vEnHpvlHXnAXZyZKfh8rKMbE30HRQGovuegeNsHmy7YpLgtZ-Khw8AKUMVEpkF5u6ljkF6SFN8UuEdD6Kz9s7O1xPBo2iKYqw", "e": "AQAB", "kty": "RSA", "kid": "rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4"}', '{"created_by": "CLI tool"}', datetime.datetime(2025, 2, 7, 3, 15, 32, 537814), datetime.datetime(2025, 2, 7, 7, 53, 55, 326120), None, 2, 2])
servicekey stdout | 2025-02-07 05:53:55,335 [99] [DEBUG] [__main__] Finished automatic refresh of service key rZ1OEw8wOnqfGw5y-8Xi3Qh30iUwITq8gbz3kvF3hB4 with new expiration 2025-02-07 07:53:55.326120
servicekey stdout | 2025-02-07 05:53:55,335 [99] [DEBUG] [data.database] Disconnecting from database.
servicekey stdout | 2025-02-07 05:53:55,335 [99] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2025-02-07 06:48:55 UTC)" executed successfully
quotaregistrysizeworker stdout | 2025-02-07 05:53:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:53:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:56.118855+00:00 (in 59.999525 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:53:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:56 UTC)" (scheduled at 2025-02-07 05:53:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:53:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:53:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:53:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:37.715878+00:00 (in 41.001850 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:56 UTC)" (scheduled at 2025-02-07 05:53:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:56,721 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:56,721 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:53:56,721 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:53:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:53:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:08.732147+00:00 (in 9.999546 seconds)
notificationworker stdout | 2025-02-07 05:53:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:08 UTC)" (scheduled at 2025-02-07 05:53:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:53:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:53:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 53, 58, 732897), True, datetime.datetime(2025, 2, 7, 5, 53, 58, 732897), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:53:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:53:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:53:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:53:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:53:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:14.225378+00:00 (in 14.999527 seconds)
quotatotalworker stdout | 2025-02-07 05:53:59,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:14 UTC)" (scheduled at 2025-02-07 05:53:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:53:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:53:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:53:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:53:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:53:59,235 [235] [DEBUG] [app] Starting request: urn:request:e87d917d-c4a3-44c8-b379-605b9edb69ba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:59,235 [238] [DEBUG] [app] Starting request: urn:request:0ec951e9-acec-4373-aea0-07337baaab34 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:53:59,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:59,237 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:53:59,238 [246] [DEBUG] [app] Starting request: urn:request:1732d76d-7107-4cfe-8eb3-b0ccec8864d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:59,238 [239] [DEBUG] [app] Starting request: urn:request:596cb0bc-7d79-4b5d-ad97-7bb1382a2852 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:53:59,238 [246] [DEBUG] [app] Ending request: urn:request:1732d76d-7107-4cfe-8eb3-b0ccec8864d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1732d76d-7107-4cfe-8eb3-b0ccec8864d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:53:59,238 [239] [DEBUG] [app] Ending request: urn:request:596cb0bc-7d79-4b5d-ad97-7bb1382a2852 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:596cb0bc-7d79-4b5d-ad97-7bb1382a2852', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:53:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:53:59,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:59,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:59,240 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:53:59,241 [236] [DEBUG] [app] Starting request: urn:request:71a11f23-58e9-4599-80be-ce5fb96e4e87 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:59,241 [237] [DEBUG] [app] Starting request: urn:request:c91cbb20-21b2-43ab-bd89-3cf1470ef033 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:53:59,242 [237] [DEBUG] [app] Ending request: urn:request:c91cbb20-21b2-43ab-bd89-3cf1470ef033 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c91cbb20-21b2-43ab-bd89-3cf1470ef033', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:53:59,242 [236] [DEBUG] [app] Ending request: urn:request:71a11f23-58e9-4599-80be-ce5fb96e4e87 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71a11f23-58e9-4599-80be-ce5fb96e4e87', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:53:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:53:59,242 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:59,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:53:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:53:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:59,242 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:53:59,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:59,242 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:53:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:59,243 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:53:59,246 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:59,246 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:59,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:53:59,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:53:59,250 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:53:59,251 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:59,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:53:59,253 [238] [DEBUG] [app] Ending request: urn:request:0ec951e9-acec-4373-aea0-07337baaab34 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ec951e9-acec-4373-aea0-07337baaab34', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:53:59,253 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:53:59,253 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.018)
gunicorn-web stdout | 2025-02-07 05:53:59,253 [235] [DEBUG] [app] Ending request: urn:request:e87d917d-c4a3-44c8-b379-605b9edb69ba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e87d917d-c4a3-44c8-b379-605b9edb69ba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:53:59,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:53:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:53:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:53:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.018 118 0.019)
exportactionlogsworker stdout | 2025-02-07 05:53:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:53:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:04.936688+00:00 (in 4.998330 seconds)
exportactionlogsworker stdout | 2025-02-07 05:53:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:59 UTC)" (scheduled at 2025-02-07 05:53:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:53:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:53:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:54:59 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:54:03,418 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
repositorygcworker stdout | 2025-02-07 05:54:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:54:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:23.420886+00:00 (in 20.001316 seconds)
repositorygcworker stdout | 2025-02-07 05:54:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:03 UTC)" (scheduled at 2025-02-07 05:54:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:54:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:54:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 3, 419798), True, datetime.datetime(2025, 2, 7, 5, 54, 3, 419798), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:54:03,427 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:54:03,427 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:54:03,427 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:03 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:54:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:54:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:33.919286+00:00 (in 29.999509 seconds)
buildlogsarchiver stdout | 2025-02-07 05:54:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:33 UTC)" (scheduled at 2025-02-07 05:54:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:54:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 54, 3, 920020), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:54:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:54:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:54:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:54:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:54:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:52.937648+00:00 (in 48.000501 seconds)
exportactionlogsworker stdout | 2025-02-07 05:54:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:04 UTC)" (scheduled at 2025-02-07 05:54:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:54:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:54:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 4, 937397), True, datetime.datetime(2025, 2, 7, 5, 54, 4, 937397), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:54:04,945 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:54:04,945 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:54:04,945 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:04 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:54:05,058 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:54:06,184 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:54:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:54:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:18.732147+00:00 (in 9.999539 seconds)
notificationworker stdout | 2025-02-07 05:54:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:18 UTC)" (scheduled at 2025-02-07 05:54:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:54:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:54:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 8, 732886), True, datetime.datetime(2025, 2, 7, 5, 54, 8, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:54:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:54:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:54:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:54:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:54:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:24.423055+00:00 (in 13.998362 seconds)
gcworker stdout | 2025-02-07 05:54:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:40 UTC)" (scheduled at 2025-02-07 05:54:10.424250+00:00)
gcworker stdout | 2025-02-07 05:54:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:54:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907350430, None, 1, 0])
gcworker stdout | 2025-02-07 05:54:10,432 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:54:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:40 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:54:10,470 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:54:11,458 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:54:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:54:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:41.831521+00:00 (in 29.999560 seconds)
autopruneworker stdout | 2025-02-07 05:54:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:41 UTC)" (scheduled at 2025-02-07 05:54:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:54:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738904051836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:54:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:54:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:54:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:54:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:54:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:29.225378+00:00 (in 14.999517 seconds)
quotatotalworker stdout | 2025-02-07 05:54:14,226 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:29 UTC)" (scheduled at 2025-02-07 05:54:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:54:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:54:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:54:14,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:54:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:54:14,235 [236] [DEBUG] [app] Starting request: urn:request:55bf1f4f-9f7e-4d85-9cdb-ef85de187303 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:14,235 [235] [DEBUG] [app] Starting request: urn:request:65abf598-89c3-4e2c-a83f-1a03cfa267b4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:14,237 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:14,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:54:14,238 [246] [DEBUG] [app] Starting request: urn:request:3a41f94f-6cb6-4cd0-b2d8-b4f275eeca11 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:14,239 [246] [DEBUG] [app] Ending request: urn:request:3a41f94f-6cb6-4cd0-b2d8-b4f275eeca11 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3a41f94f-6cb6-4cd0-b2d8-b4f275eeca11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:54:14,239 [245] [DEBUG] [app] Starting request: urn:request:e13f975a-0369-4407-ac84-684a2799a917 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-registry stdout | 2025-02-07 05:54:14,239 [245] [DEBUG] [app] Ending request: urn:request:e13f975a-0369-4407-ac84-684a2799a917 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e13f975a-0369-4407-ac84-684a2799a917', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:54:14,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:54:14,240 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:54:14,240 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:14,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:14,241 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:14,242 [238] [DEBUG] [app] Starting request: urn:request:2bb9abc9-31c4-416e-a03d-21733f8bd60c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:14,242 [238] [DEBUG] [app] Ending request: urn:request:2bb9abc9-31c4-416e-a03d-21733f8bd60c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2bb9abc9-31c4-416e-a03d-21733f8bd60c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:54:14,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:54:14,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:14,243 [237] [DEBUG] [app] Starting request: urn:request:58c1d41e-8440-41eb-916e-36cd07f5dba2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:14,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:14,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:14,243 [237] [DEBUG] [app] Ending request: urn:request:58c1d41e-8440-41eb-916e-36cd07f5dba2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:58c1d41e-8440-41eb-916e-36cd07f5dba2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:54:14,244 [237] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:54:14,244 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:14,244 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:14,244 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:14,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:14,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:14,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:14,248 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:14,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:14,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:14,252 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:14,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:14,254 [236] [DEBUG] [app] Ending request: urn:request:55bf1f4f-9f7e-4d85-9cdb-ef85de187303 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55bf1f4f-9f7e-4d85-9cdb-ef85de187303', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:14,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:14,254 [235] [DEBUG] [app] Ending request: urn:request:65abf598-89c3-4e2c-a83f-1a03cfa267b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:65abf598-89c3-4e2c-a83f-1a03cfa267b4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:14,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:14,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:54:14,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
quotatotalworker stdout | 2025-02-07 05:54:15,579 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:54:16,143 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:54:16,586 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:54:18,039 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:54:18,165 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:54:18,168 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:54:18,171 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:54:18,174 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:54:18,176 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:54:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:54:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:28.732147+00:00 (in 9.999543 seconds)
notificationworker stdout | 2025-02-07 05:54:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:28 UTC)" (scheduled at 2025-02-07 05:54:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:54:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:54:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 18, 732886), True, datetime.datetime(2025, 2, 7, 5, 54, 18, 732886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:54:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:54:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:54:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:28 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:54:18,774 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:54:18,846 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-secscan stdout | 2025-02-07 05:54:19,178 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:54:19,181 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:54:19,183 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:54:20,153 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:20,364 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-registry stdout | 2025-02-07 05:54:20,679 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:54:20,684 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:54:20,689 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:54:20,692 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:54:20,694 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:54:20,697 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:54:20,701 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:54:20,704 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:54:20,706 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
servicekey stdout | 2025-02-07 05:54:20,773 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
globalpromstats stdout | 2025-02-07 05:54:20,922 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:54:21,162 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:54:21,580 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:54:21,918 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:54:22,181 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:54:22,295 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
repositorygcworker stdout | 2025-02-07 05:54:23,421 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:54:23,421 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:03.419110+00:00 (in 39.997743 seconds)
repositorygcworker stdout | 2025-02-07 05:54:23,421 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:23 UTC)" (scheduled at 2025-02-07 05:54:23.420886+00:00)
repositorygcworker stdout | 2025-02-07 05:54:23,421 [93] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2025-02-07 05:54:23,421 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:23 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:54:23,947 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:54:23,978 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gcworker stdout | 2025-02-07 05:54:24,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:54:24,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:40.424250+00:00 (in 16.000750 seconds)
gcworker stdout | 2025-02-07 05:54:24,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:54 UTC)" (scheduled at 2025-02-07 05:54:24.423055+00:00)
gcworker stdout | 2025-02-07 05:54:24,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:54:24,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:54:54 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:54:24,487 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
notificationworker stdout | 2025-02-07 05:54:28,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:54:28,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:38.732147+00:00 (in 9.999573 seconds)
notificationworker stdout | 2025-02-07 05:54:28,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:38 UTC)" (scheduled at 2025-02-07 05:54:28.732147+00:00)
notificationworker stdout | 2025-02-07 05:54:28,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:54:28,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 28, 732849), True, datetime.datetime(2025, 2, 7, 5, 54, 28, 732849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:54:28,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:54:28,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:54:28,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:38 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:54:29,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:54:29,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:44.225378+00:00 (in 14.999544 seconds)
quotatotalworker stdout | 2025-02-07 05:54:29,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:44 UTC)" (scheduled at 2025-02-07 05:54:29.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:54:29,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:54:29,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:54:29,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:54:29,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:44 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:54:29,235 [237] [DEBUG] [app] Starting request: urn:request:7cfeb528-4acc-433f-b5e1-96bdcdbfe442 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:29,235 [235] [DEBUG] [app] Starting request: urn:request:f70497a1-daf0-4d79-a4b0-6a7e2d732a74 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:29,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:29,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:54:29,238 [245] [DEBUG] [app] Starting request: urn:request:d1cf25e7-fff4-4770-808b-3ddef7d1fd57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:29,238 [239] [DEBUG] [app] Starting request: urn:request:cd6a5c0f-879f-44ce-b170-244f28f4e659 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:29,239 [239] [DEBUG] [app] Ending request: urn:request:cd6a5c0f-879f-44ce-b170-244f28f4e659 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cd6a5c0f-879f-44ce-b170-244f28f4e659', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:54:29,239 [245] [DEBUG] [app] Ending request: urn:request:d1cf25e7-fff4-4770-808b-3ddef7d1fd57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d1cf25e7-fff4-4770-808b-3ddef7d1fd57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:54:29,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:54:29,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:29 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:54:29,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:29,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:29,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:29,240 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:29,241 [235] [DEBUG] [app] Starting request: urn:request:136d0f71-9af6-42b5-9d8d-a1a45ec01fce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:29,241 [238] [DEBUG] [app] Starting request: urn:request:e7636ef3-e9d9-4a43-a9e3-d519f3fe32e5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:29,241 [235] [DEBUG] [app] Ending request: urn:request:136d0f71-9af6-42b5-9d8d-a1a45ec01fce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:136d0f71-9af6-42b5-9d8d-a1a45ec01fce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:54:29,242 [238] [DEBUG] [app] Ending request: urn:request:e7636ef3-e9d9-4a43-a9e3-d519f3fe32e5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e7636ef3-e9d9-4a43-a9e3-d519f3fe32e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:54:29,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:54:29,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:29,242 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:29 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:29 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:54:29,242 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:29,242 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:29,242 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:29,242 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:29,243 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:29,246 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:29,246 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:29,246 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:29,246 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:29,250 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:29,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:29,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:29,252 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:29,253 [235] [DEBUG] [app] Ending request: urn:request:f70497a1-daf0-4d79-a4b0-6a7e2d732a74 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f70497a1-daf0-4d79-a4b0-6a7e2d732a74', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:29,253 [237] [DEBUG] [app] Ending request: urn:request:7cfeb528-4acc-433f-b5e1-96bdcdbfe442 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7cfeb528-4acc-433f-b5e1-96bdcdbfe442', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:29,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:29,253 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:29,254 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:54:29,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:29 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:29 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
namespacegcworker stdout | 2025-02-07 05:54:31,621 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:54:31,621 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:32.625945+00:00 (in 1.004326 seconds)
namespacegcworker stdout | 2025-02-07 05:54:31,621 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:31 UTC)" (scheduled at 2025-02-07 05:54:31.621131+00:00)
namespacegcworker stdout | 2025-02-07 05:54:31,621 [71] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2025-02-07 05:54:31,622 [71] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 31, 621931), True, datetime.datetime(2025, 2, 7, 5, 54, 31, 621931), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2025-02-07 05:54:31,629 [71] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2025-02-07 05:54:31,629 [71] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2025-02-07 05:54:31,629 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:31 UTC)" executed successfully
namespacegcworker stdout | 2025-02-07 05:54:32,626 [71] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2025-02-07 05:54:32,626 [71] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:31.621131+00:00 (in 58.994620 seconds)
namespacegcworker stdout | 2025-02-07 05:54:32,626 [71] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:32 UTC)" (scheduled at 2025-02-07 05:54:32.625945+00:00)
namespacegcworker stdout | 2025-02-07 05:54:32,626 [71] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2025-02-07 05:54:32,626 [71] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:32 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:54:33,434 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:54:33,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:54:33,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:03.919286+00:00 (in 29.999569 seconds)
buildlogsarchiver stdout | 2025-02-07 05:54:33,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:03 UTC)" (scheduled at 2025-02-07 05:54:33.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:54:33,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 54, 33, 919983), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:54:33,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:54:33,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:54:33,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:03 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:54:35,088 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:54:36,211 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:37,716 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:37,716 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:56.713547+00:00 (in 18.997201 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:37,716 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:37 UTC)" (scheduled at 2025-02-07 05:54:37.715878+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:37,717 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."artifact_type_backfilled" = %s) OR ("t1"."artifact_type_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:37,723 [69] [DEBUG] [__main__] Manifest artifact_type backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:37,723 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:37,723 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_artifact_type (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:37 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:54:38,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:54:38,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:48.732147+00:00 (in 9.999571 seconds)
notificationworker stdout | 2025-02-07 05:54:38,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:48 UTC)" (scheduled at 2025-02-07 05:54:38.732147+00:00)
notificationworker stdout | 2025-02-07 05:54:38,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:54:38,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 38, 732843), True, datetime.datetime(2025, 2, 7, 5, 54, 38, 732843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:54:38,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:54:38,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:54:38,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:48 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:54:40,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:54:40,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:54.423055+00:00 (in 13.998380 seconds)
gcworker stdout | 2025-02-07 05:54:40,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:10 UTC)" (scheduled at 2025-02-07 05:54:40.424250+00:00)
gcworker stdout | 2025-02-07 05:54:40,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:54:40,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907380430, None, 1, 0])
gcworker stdout | 2025-02-07 05:54:40,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:54:40,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:10 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:54:40,506 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:54:41,494 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:54:41,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:54:41,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:11.831521+00:00 (in 29.999571 seconds)
autopruneworker stdout | 2025-02-07 05:54:41,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:11 UTC)" (scheduled at 2025-02-07 05:54:41.831521+00:00)
autopruneworker stdout | 2025-02-07 05:54:41,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738904081836, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:54:41,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:54:41,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:54:41,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:11 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:54:44,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:54:44,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:59.225378+00:00 (in 14.999563 seconds)
quotatotalworker stdout | 2025-02-07 05:54:44,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:59 UTC)" (scheduled at 2025-02-07 05:54:44.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:54:44,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:54:44,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:54:44,232 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:54:44,232 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:54:59 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:54:44,236 [235] [DEBUG] [app] Starting request: urn:request:498810da-f465-4fb8-aa77-2aaef946678d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:44,236 [237] [DEBUG] [app] Starting request: urn:request:2df6a1d5-fd70-4ac2-ab21-99936f99e3d0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:44,237 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:44,237 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:54:44,239 [246] [DEBUG] [app] Starting request: urn:request:e4a2e26c-b006-4ab4-af76-bf6109a09786 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:44,239 [245] [DEBUG] [app] Starting request: urn:request:977e0b56-24ab-4726-88d9-f20057d2ea8c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:44,239 [246] [DEBUG] [app] Ending request: urn:request:e4a2e26c-b006-4ab4-af76-bf6109a09786 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e4a2e26c-b006-4ab4-af76-bf6109a09786', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:54:44,239 [245] [DEBUG] [app] Ending request: urn:request:977e0b56-24ab-4726-88d9-f20057d2ea8c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:977e0b56-24ab-4726-88d9-f20057d2ea8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:54:44,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:54:44,239 [245] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:44 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 162 0.001)
gunicorn-web stdout | 2025-02-07 05:54:44,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:44,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:44,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:44,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:44,242 [236] [DEBUG] [app] Starting request: urn:request:795c81cb-e965-4102-8c65-0651e886f5e1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:44,242 [236] [DEBUG] [app] Ending request: urn:request:795c81cb-e965-4102-8c65-0651e886f5e1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:795c81cb-e965-4102-8c65-0651e886f5e1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:54:44,242 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:54:44,243 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:44,242 [238] [DEBUG] [app] Starting request: urn:request:31d22ebf-eb36-43d9-ace9-d0d08e5ae38f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:44,243 [238] [DEBUG] [app] Ending request: urn:request:31d22ebf-eb36-43d9-ace9-d0d08e5ae38f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:31d22ebf-eb36-43d9-ace9-d0d08e5ae38f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:54:44,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:44,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:44,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:44 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:54:44,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:44 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:54:44,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:44,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:44,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:44,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:44,247 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:44,247 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:44,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:44,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:44,252 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:44,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:44,253 [235] [DEBUG] [app] Ending request: urn:request:498810da-f465-4fb8-aa77-2aaef946678d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:498810da-f465-4fb8-aa77-2aaef946678d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:44,253 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:44,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.019 118 0.019)
gunicorn-web stdout | 2025-02-07 05:54:44,254 [237] [DEBUG] [app] Ending request: urn:request:2df6a1d5-fd70-4ac2-ab21-99936f99e3d0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2df6a1d5-fd70-4ac2-ab21-99936f99e3d0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:44,254 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:44,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:44 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:44 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
quotatotalworker stdout | 2025-02-07 05:54:45,616 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:54:46,180 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:54:46,622 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:54:48,058 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:54:48,174 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:54:48,177 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:54:48,180 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:54:48,182 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:54:48,185 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:54:48,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:54:48,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:50.814677+00:00 (in 2.082118 seconds)
notificationworker stdout | 2025-02-07 05:54:48,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:58 UTC)" (scheduled at 2025-02-07 05:54:48.732147+00:00)
notificationworker stdout | 2025-02-07 05:54:48,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:54:48,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 48, 732837), True, datetime.datetime(2025, 2, 7, 5, 54, 48, 732837), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:54:48,739 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:54:48,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:54:48,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:54:58 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:54:48,804 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:54:48,861 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-secscan stdout | 2025-02-07 05:54:49,187 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:54:49,190 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:54:49,192 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}
autopruneworker stdout | 2025-02-07 05:54:50,181 [54] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'autopruneworker.py', 'pid': '54'}
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:50,401 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestsubjectbackfillworker.py', 'pid': '69'}
gunicorn-registry stdout | 2025-02-07 05:54:50,692 [239] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '239'}
gunicorn-registry stdout | 2025-02-07 05:54:50,695 [244] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '244'}
gunicorn-registry stdout | 2025-02-07 05:54:50,697 [245] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '245'}
gunicorn-registry stdout | 2025-02-07 05:54:50,700 [246] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '246'}
gunicorn-registry stdout | 2025-02-07 05:54:50,703 [64] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '64'}
gunicorn-registry stdout | 2025-02-07 05:54:50,705 [241] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '241'}
gunicorn-registry stdout | 2025-02-07 05:54:50,710 [243] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '243'}
gunicorn-registry stdout | 2025-02-07 05:54:50,713 [242] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '242'}
gunicorn-registry stdout | 2025-02-07 05:54:50,715 [240] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'registry:application', 'pid': '240'}
servicekey stdout | 2025-02-07 05:54:50,781 [99] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'servicekeyworker.py', 'pid': '99'}
notificationworker stdout | 2025-02-07 05:54:50,814 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:54:50,815 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:58.732147+00:00 (in 7.917013 seconds)
notificationworker stdout | 2025-02-07 05:54:50,815 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:50 UTC)" (scheduled at 2025-02-07 05:54:50.814677+00:00)
notificationworker stdout | 2025-02-07 05:54:50,815 [73] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2025-02-07 05:54:50,815 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:50 UTC)" executed successfully
globalpromstats stdout | 2025-02-07 05:54:50,958 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'globalpromstats.py', 'pid': '63'}
exportactionlogsworker stdout | 2025-02-07 05:54:51,198 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'exportactionlogsworker.py', 'pid': '61'}
repositorygcworker stdout | 2025-02-07 05:54:51,617 [93] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositorygcworker.py', 'pid': '93'}
gcworker stdout | 2025-02-07 05:54:51,954 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'gcworker.py', 'pid': '62'}
manifestbackfillworker stdout | 2025-02-07 05:54:52,217 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'manifestbackfillworker.py', 'pid': '68'}
securityscanningnotificationworker stdout | 2025-02-07 05:54:52,329 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'securityscanningnotificationworker.py', 'pid': '97'}
exportactionlogsworker stdout | 2025-02-07 05:54:52,937 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:54:52,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:54:59.937899+00:00 (in 6.999753 seconds)
exportactionlogsworker stdout | 2025-02-07 05:54:52,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:59:52 UTC)" (scheduled at 2025-02-07 05:54:52.937648+00:00)
exportactionlogsworker stdout | 2025-02-07 05:54:52,938 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2025, 2, 7, 5, 54, 52, 938345), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:54:52,945 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 54, 52, 938345), True, datetime.datetime(2025, 2, 7, 5, 54, 52, 938345), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:54:52,947 [61] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2025, 2, 7, 5, 54, 52, 938345), True, datetime.datetime(2025, 2, 7, 5, 54, 52, 938345), 0, 'exportactionlogs/%', False, datetime.datetime(2025, 2, 7, 5, 54, 52, 938345), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2025-02-07 05:54:52,948 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:54:52,948 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2025-02-07 05:59:52 UTC)" executed successfully
buildlogsarchiver stdout | 2025-02-07 05:54:53,968 [57] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'buildlogsarchiver.py', 'pid': '57'}
namespacegcworker stdout | 2025-02-07 05:54:54,015 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'namespacegcworker.py', 'pid': '71'}
gcworker stdout | 2025-02-07 05:54:54,423 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:54:54,423 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:10.424250+00:00 (in 16.000760 seconds)
gcworker stdout | 2025-02-07 05:54:54,423 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:24 UTC)" (scheduled at 2025-02-07 05:54:54.423055+00:00)
gcworker stdout | 2025-02-07 05:54:54,423 [62] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2025-02-07 05:54:54,423 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:24 UTC)" executed successfully
blobuploadcleanupworker stdout | 2025-02-07 05:54:54,523 [55] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'blobuploadcleanupworker.py', 'pid': '55'}
quotaregistrysizeworker stdout | 2025-02-07 05:54:56,119 [76] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2025-02-07 05:54:56,119 [76] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:56.118855+00:00 (in 59.999546 seconds)
quotaregistrysizeworker stdout | 2025-02-07 05:54:56,119 [76] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:56 UTC)" (scheduled at 2025-02-07 05:54:56.118855+00:00)
quotaregistrysizeworker stdout | 2025-02-07 05:54:56,119 [76] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2025-02-07 05:54:56,125 [76] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2025-02-07 05:54:56,125 [76] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:56 UTC)" executed successfully
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:56,713 [69] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:56,714 [69] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:37.715878+00:00 (in 41.001887 seconds)
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:56,714 [69] [INFO] [apscheduler.executors.default] Running job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:56 UTC)" (scheduled at 2025-02-07 05:54:56.713547+00:00)
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:56,714 [69] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t1"."subject", "t1"."subject_backfilled", "t1"."artifact_type", "t1"."artifact_type_backfilled" FROM "manifest" AS "t1" WHERE (("t1"."subject_backfilled" = %s) OR ("t1"."subject_backfilled" IS %s)) LIMIT %s OFFSET %s', [False, None, 1, 0])
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:56,720 [69] [DEBUG] [__main__] Manifest subject backfill worker has completed; skipping
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:56,720 [69] [DEBUG] [data.database] Disconnecting from database.
manifestsubjectbackfillworker stdout | 2025-02-07 05:54:56,720 [69] [INFO] [apscheduler.executors.default] Job "ManifestSubjectBackfillWorker._backfill_manifest_subject (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:56 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:54:58,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:54:58,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:08.732147+00:00 (in 9.999516 seconds)
notificationworker stdout | 2025-02-07 05:54:58,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:55:08 UTC)" (scheduled at 2025-02-07 05:54:58.732147+00:00)
notificationworker stdout | 2025-02-07 05:54:58,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:54:58,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 54, 58, 732914), True, datetime.datetime(2025, 2, 7, 5, 54, 58, 732914), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:54:58,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:54:58,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:54:58,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:55:08 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:54:59,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:54:59,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:14.225378+00:00 (in 14.999554 seconds)
quotatotalworker stdout | 2025-02-07 05:54:59,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:55:14 UTC)" (scheduled at 2025-02-07 05:54:59.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:54:59,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:54:59,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:54:59,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:54:59,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:55:14 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:54:59,235 [236] [DEBUG] [app] Starting request: urn:request:bac09ced-8f5b-4e6e-8344-c22a6e35a990 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:59,235 [235] [DEBUG] [app] Starting request: urn:request:f1001833-8ba3-4ebf-9cc8-b3962fdfd9c6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:54:59,236 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:59,236 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:54:59,238 [242] [DEBUG] [app] Starting request: urn:request:71071172-0d3d-4f31-93dc-3faa910a1268 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:59,238 [246] [DEBUG] [app] Starting request: urn:request:ab5e7d0c-342c-4555-9fd9-30ad59b4de52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:54:59,238 [242] [DEBUG] [app] Ending request: urn:request:71071172-0d3d-4f31-93dc-3faa910a1268 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:71071172-0d3d-4f31-93dc-3faa910a1268', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:54:59,238 [246] [DEBUG] [app] Ending request: urn:request:ab5e7d0c-342c-4555-9fd9-30ad59b4de52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab5e7d0c-342c-4555-9fd9-30ad59b4de52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-registry stdout | 2025-02-07 05:54:59,239 [242] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:54:59,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:59 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:54:59,239 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:59,239 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:59,240 [236] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:59,240 [235] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:54:59,242 [235] [DEBUG] [app] Starting request: urn:request:121b9c10-a9e0-4b17-9394-67d7375065d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:59,242 [238] [DEBUG] [app] Starting request: urn:request:392b8e56-d4d2-400a-9b57-af8897f493be (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:54:59,242 [235] [DEBUG] [app] Ending request: urn:request:121b9c10-a9e0-4b17-9394-67d7375065d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:121b9c10-a9e0-4b17-9394-67d7375065d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:54:59,242 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:54:59,242 [238] [DEBUG] [app] Ending request: urn:request:392b8e56-d4d2-400a-9b57-af8897f493be (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:392b8e56-d4d2-400a-9b57-af8897f493be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:54:59,242 [235] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:54:59 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 159 0.002)
gunicorn-web stdout | 2025-02-07 05:54:59,243 [238] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:54:59 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:54:59,243 [236] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:54:59,243 [235] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:59,243 [235] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:59,243 [236] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:54:59,243 [236] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:54:59,247 [236] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:59,247 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:59,247 [235] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:54:59,247 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:54:59,251 [236] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:59,251 [235] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:54:59,253 [235] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:59,253 [236] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:54:59,254 [235] [DEBUG] [app] Ending request: urn:request:f1001833-8ba3-4ebf-9cc8-b3962fdfd9c6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1001833-8ba3-4ebf-9cc8-b3962fdfd9c6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:59,254 [236] [DEBUG] [app] Ending request: urn:request:bac09ced-8f5b-4e6e-8344-c22a6e35a990 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bac09ced-8f5b-4e6e-8344-c22a6e35a990', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:54:59,254 [235] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:59,254 [236] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:54:59,254 [236] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:54:59,254 [235] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:54:59 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:54:59 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.020 118 0.020)
exportactionlogsworker stdout | 2025-02-07 05:54:59,938 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:54:59,938 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:04.936688+00:00 (in 4.998277 seconds)
exportactionlogsworker stdout | 2025-02-07 05:54:59,938 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:59 UTC)" (scheduled at 2025-02-07 05:54:59.937899+00:00)
exportactionlogsworker stdout | 2025-02-07 05:54:59,938 [61] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2025-02-07 05:54:59,938 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2025-02-07 05:55:59 UTC)" executed successfully
repositorygcworker stdout | 2025-02-07 05:55:03,419 [93] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2025-02-07 05:55:03,419 [93] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:23.420886+00:00 (in 20.001336 seconds)
repositorygcworker stdout | 2025-02-07 05:55:03,419 [93] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:56:03 UTC)" (scheduled at 2025-02-07 05:55:03.419110+00:00)
repositorygcworker stdout | 2025-02-07 05:55:03,419 [93] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2025-02-07 05:55:03,420 [93] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 55, 3, 419757), True, datetime.datetime(2025, 2, 7, 5, 55, 3, 419757), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2025-02-07 05:55:03,426 [93] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2025-02-07 05:55:03,426 [93] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2025-02-07 05:55:03,426 [93] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:56:03 UTC)" executed successfully
storagereplication stdout | 2025-02-07 05:55:03,461 [100] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'storagereplication.py', 'pid': '100'}
buildlogsarchiver stdout | 2025-02-07 05:55:03,919 [57] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2025-02-07 05:55:03,919 [57] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:33.919286+00:00 (in 29.999565 seconds)
buildlogsarchiver stdout | 2025-02-07 05:55:03,919 [57] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:33 UTC)" (scheduled at 2025-02-07 05:55:03.919286+00:00)
buildlogsarchiver stdout | 2025-02-07 05:55:03,920 [57] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2025, 1, 23, 5, 55, 3, 919906), False, 50, 1, 0])
buildlogsarchiver stdout | 2025-02-07 05:55:03,927 [57] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2025-02-07 05:55:03,927 [57] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2025-02-07 05:55:03,927 [57] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:33 UTC)" executed successfully
exportactionlogsworker stdout | 2025-02-07 05:55:04,936 [61] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2025-02-07 05:55:04,937 [61] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:59.937899+00:00 (in 55.000764 seconds)
exportactionlogsworker stdout | 2025-02-07 05:55:04,937 [61] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:56:04 UTC)" (scheduled at 2025-02-07 05:55:04.936688+00:00)
exportactionlogsworker stdout | 2025-02-07 05:55:04,937 [61] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2025-02-07 05:55:04,938 [61] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 55, 4, 937453), True, datetime.datetime(2025, 2, 7, 5, 55, 4, 937453), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2025-02-07 05:55:04,944 [61] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2025-02-07 05:55:04,944 [61] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2025-02-07 05:55:04,944 [61] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2025-02-07 05:56:04 UTC)" executed successfully
repositoryactioncounter stdout | 2025-02-07 05:55:05,117 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'repositoryactioncounter.py', 'pid': '84'}
queuecleanupworker stdout | 2025-02-07 05:55:06,228 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'queuecleanupworker.py', 'pid': '75'}
notificationworker stdout | 2025-02-07 05:55:08,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:55:08,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:18.732147+00:00 (in 9.999536 seconds)
notificationworker stdout | 2025-02-07 05:55:08,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:55:18 UTC)" (scheduled at 2025-02-07 05:55:08.732147+00:00)
notificationworker stdout | 2025-02-07 05:55:08,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:55:08,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 55, 8, 732922), True, datetime.datetime(2025, 2, 7, 5, 55, 8, 732922), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:55:08,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:55:08,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:55:08,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:55:18 UTC)" executed successfully
gcworker stdout | 2025-02-07 05:55:10,424 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2025-02-07 05:55:10,424 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:24.423055+00:00 (in 13.998383 seconds)
gcworker stdout | 2025-02-07 05:55:10,424 [62] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:40 UTC)" (scheduled at 2025-02-07 05:55:10.424250+00:00)
gcworker stdout | 2025-02-07 05:55:10,425 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['repo_image_expiry', 1, 0])
gcworker stdout | 2025-02-07 05:55:10,431 [62] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."method_id", "t1"."repository_id", "t1"."event_config_json" FROM "repositorynotification" AS "t1" WHERE ((("t1"."event_id" = %s) AND ("t1"."number_of_failures" < %s)) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [11, 3, 1738907410431, None, 1, 0])
gcworker stdout | 2025-02-07 05:55:10,433 [62] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2025-02-07 05:55:10,433 [62] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._scan_notifications (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:40 UTC)" executed successfully
teamsyncworker stdout | 2025-02-07 05:55:10,539 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotaregistrysizeworker stdout | 2025-02-07 05:55:11,530 [76] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotaregistrysizeworker.py', 'pid': '76'}
autopruneworker stdout | 2025-02-07 05:55:11,831 [54] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2025-02-07 05:55:11,831 [54] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:41.831521+00:00 (in 29.999567 seconds)
autopruneworker stdout | 2025-02-07 05:55:11,832 [54] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:41 UTC)" (scheduled at 2025-02-07 05:55:11.831521+00:00)
autopruneworker stdout | 2025-02-07 05:55:11,836 [54] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1738904111835, None, 1, 0])
autopruneworker stdout | 2025-02-07 05:55:11,839 [54] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2025-02-07 05:55:11,839 [54] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2025-02-07 05:55:11,839 [54] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2025-02-07 05:55:41 UTC)" executed successfully
quotatotalworker stdout | 2025-02-07 05:55:14,225 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2025-02-07 05:55:14,225 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:29.225378+00:00 (in 14.999541 seconds)
quotatotalworker stdout | 2025-02-07 05:55:14,225 [77] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:55:29 UTC)" (scheduled at 2025-02-07 05:55:14.225378+00:00)
quotatotalworker stdout | 2025-02-07 05:55:14,226 [77] [INFO] [__main__] Quota backfill worker started, searching for namespaces to calculate size
quotatotalworker stdout | 2025-02-07 05:55:14,226 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2025-02-07 05:55:14,233 [77] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2025-02-07 05:55:14,233 [77] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2025-02-07 05:55:29 UTC)" executed successfully
gunicorn-web stdout | 2025-02-07 05:55:14,235 [237] [DEBUG] [app] Starting request: urn:request:643bb114-e6ed-40c5-9b46-0871452125cb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:55:14,235 [238] [DEBUG] [app] Starting request: urn:request:54d88df7-4604-406b-aa0a-4d69c944e83d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2025-02-07 05:55:14,236 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:55:14,236 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2025-02-07 05:55:14,238 [239] [DEBUG] [app] Starting request: urn:request:0214524f-29d4-4996-8ccd-2cbec1f8a97c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:55:14,238 [246] [DEBUG] [app] Starting request: urn:request:ee096ce2-6458-4c60-8b7c-694ee7607c0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2025-02-07 05:55:14,238 [246] [DEBUG] [app] Ending request: urn:request:ee096ce2-6458-4c60-8b7c-694ee7607c0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee096ce2-6458-4c60-8b7c-694ee7607c0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:55:14,238 [239] [DEBUG] [app] Ending request: urn:request:0214524f-29d4-4996-8ccd-2cbec1f8a97c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0214524f-29d4-4996-8ccd-2cbec1f8a97c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '1f9c23ce', 'user-agent': 'python-requests/2.32.2'}
gunicorn-registry stdout | 2025-02-07 05:55:14,239 [246] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
gunicorn-registry stdout | 2025-02-07 05:55:14,239 [239] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:55:14 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.002 162 0.002)
gunicorn-web stdout | 2025-02-07 05:55:14,239 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:55:14,239 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:55:14,241 [237] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:55:14,241 [238] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2025-02-07 05:55:14,242 [235] [DEBUG] [app] Starting request: urn:request:d4021343-b400-40d2-a6a5-a92dfaf88c0d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:55:14,242 [236] [DEBUG] [app] Starting request: urn:request:d8942dcd-9062-4ccf-abad-d9691c9ecdb5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2025-02-07 05:55:14,243 [235] [DEBUG] [app] Ending request: urn:request:d4021343-b400-40d2-a6a5-a92dfaf88c0d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4021343-b400-40d2-a6a5-a92dfaf88c0d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:55:14,243 [236] [DEBUG] [app] Ending request: urn:request:d8942dcd-9062-4ccf-abad-d9691c9ecdb5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d8942dcd-9062-4ccf-abad-d9691c9ecdb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'python-requests/2.32.2'}
gunicorn-web stdout | 2025-02-07 05:55:14,243 [235] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:55:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:55:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [07/Feb/2025:05:55:14 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.32.2" (0.001 159 0.001)
gunicorn-web stdout | 2025-02-07 05:55:14,243 [236] [INFO] [gunicorn.access] 127.0.0.1 - - [07/Feb/2025:05:55:14 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.32.2"
gunicorn-web stdout | 2025-02-07 05:55:14,243 [237] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:55:14,243 [238] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2025-02-07 05:55:14,244 [238] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:55:14,244 [237] [DEBUG] [data.model.health] Validating database connection.
gunicorn-web stdout | 2025-02-07 05:55:14,244 [237] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:55:14,244 [238] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2025-02-07 05:55:14,248 [238] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:55:14,248 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:55:14,248 [237] [DEBUG] [data.model.health] Checking for existence of team roles, timeout 5000 ms.
gunicorn-web stdout | 2025-02-07 05:55:14,248 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (5000,))
gunicorn-web stdout | 2025-02-07 05:55:14,252 [238] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:55:14,252 [237] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2025-02-07 05:55:14,253 [238] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:55:14,253 [237] [DEBUG] [peewee] ('SET statement_timeout=%s;', (0,))
gunicorn-web stdout | 2025-02-07 05:55:14,255 [238] [DEBUG] [app] Ending request: urn:request:54d88df7-4604-406b-aa0a-4d69c944e83d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:54d88df7-4604-406b-aa0a-4d69c944e83d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:55:14,255 [237] [DEBUG] [app] Ending request: urn:request:643bb114-e6ed-40c5-9b46-0871452125cb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:643bb114-e6ed-40c5-9b46-0871452125cb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.36/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': '199aa909', 'user-agent': 'kube-probe/1.30'}
gunicorn-web stdout | 2025-02-07 05:55:14,255 [238] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:55:14,255 [237] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2025-02-07 05:55:14,255 [238] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:55:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
gunicorn-web stdout | 2025-02-07 05:55:14,255 [237] [INFO] [gunicorn.access] 10.129.2.2 - - [07/Feb/2025:05:55:14 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.30"
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:55:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
nginx stdout | 10.129.2.2 (-) - - [07/Feb/2025:05:55:14 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.30" (0.021 118 0.021)
quotatotalworker stdout | 2025-02-07 05:55:15,643 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'quotatotalworker.py', 'pid': '77'}
builder stdout | 2025-02-07 05:55:16,218 [56] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'builder.py', 'pid': '56'}
logrotateworker stdout | 2025-02-07 05:55:16,652 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'logrotateworker.py', 'pid': '67'}
expiredappspecifictokenworker stdout | 2025-02-07 05:55:18,095 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '60'}
gunicorn-web stdout | 2025-02-07 05:55:18,182 [235] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '235'}
gunicorn-web stdout | 2025-02-07 05:55:18,187 [236] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '236'}
gunicorn-web stdout | 2025-02-07 05:55:18,190 [237] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '237'}
gunicorn-web stdout | 2025-02-07 05:55:18,194 [238] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '238'}
gunicorn-web stdout | 2025-02-07 05:55:18,197 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'web:application', 'pid': '66'}
notificationworker stdout | 2025-02-07 05:55:18,732 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2025-02-07 05:55:18,732 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2025-02-07 05:55:28.732147+00:00 (in 9.999525 seconds)
notificationworker stdout | 2025-02-07 05:55:18,732 [73] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:55:28 UTC)" (scheduled at 2025-02-07 05:55:18.732147+00:00)
notificationworker stdout | 2025-02-07 05:55:18,732 [73] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2025-02-07 05:55:18,733 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2025, 2, 7, 5, 55, 18, 732892), True, datetime.datetime(2025, 2, 7, 5, 55, 18, 732892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2025-02-07 05:55:18,740 [73] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2025-02-07 05:55:18,740 [73] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2025-02-07 05:55:18,740 [73] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2025-02-07 05:55:28 UTC)" executed successfully
notificationworker stdout | 2025-02-07 05:55:18,818 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'notificationworker.py', 'pid': '73'}
chunkcleanupworker stdout | 2025-02-07 05:55:18,882 [58] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'chunkcleanupworker.py', 'pid': '58'}
gunicorn-secscan stdout | 2025-02-07 05:55:19,196 [234] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '234'}
gunicorn-secscan stdout | 2025-02-07 05:55:19,199 [233] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '233'}
gunicorn-secscan stdout | 2025-02-07 05:55:19,202 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-56fc558958-nsfkn', 'process_name': 'secscan:application', 'pid': '65'}